Overview of Multilingual Opinion Analysis Task at NTCIR-7
Yohei Seki†, David Kirk Evans‡, Lun-Wei Ku§,
Le Sun¶, Hsin-Hsi Chen§, Noriko Kando‡†Dept. of Information and Computer Sciences, Toyohashi University of Technology
Aichi 441-8580, Japanseki@ics.tut.ac.jp‡National Institute of Informatics
Tokyo 101-8430, Japan{devans, kando}@nii.ac.jp§Dept. of Computer Science and Information Engineering, National Taiwan University
Taipei 10617, Taiwan{lwku, hhchen}@csie.ntu.edu.tw¶Institute of Software, Chinese Academy of Sciences
Beijing 100080, P.R. China
sunle@iscas.cn
November 24, 2008

Abstract
This paper describes an overview of the Multilingual Opinion Analysis Task from 2007 to 2008 at
the Seventh NTCIR Workshop. We created testcollections of 22, 17, 17, 16 topics (7,163, 4,711,
6,174, and 5,301 sentences) in Japanese, English,
Traditional Chinese, and Simpliﬁed Chinese. Using this test collection, we conducted ﬁve subtasks: (1) mandatory opinionated sentence judgment, and optional subtasks of (2) relevant sentence judgment, (3) polarity judgment, (4) opinion holder extraction, and (5) opinion target extraction. 32 results were submitted from 21 participants with ﬁve results submitted by the organizers. In this paper we present the task deﬁnition,
the details of the test collection, the evaluation results of the groups that participated in this task,
and their approach.
Keywords: Multilingual Opinion Analysis Task
(MOAT), Opinionated Sentence, Opinion Holder,
Opinion Target, Relevance, Polarity, and Opinion
Expression Level Annotation and Evaluation.

1

Introduction

This paper describes an overview of the Multilingual Opinion Analysis Task from 2007 to 2008
at the Seventh NTCIR Workshop [4] (NTCIR-7
Opinion). This was the second eﬀort to produce
a multi-lingual test collection for evaluating opin-

ion extraction at NTCIR, following the NTCIR-6
Opinion Analysis Pilot Task [9].
Opinion and sentiment analysis has been receiving a lot of attention in the natural language
processing research community recently [6, 2, 10].
In TREC Blog track [5], opinion ﬁnding task
was conducted in 2007 and 2008. With thebroad range of information sources available on
the web, and rapid increase in the uptake of social community-oriented websites that foster usergenerated content [11], there has been further interest by both commercial and governmental parties in trying to automatically analyze and monitor the tide of prevalent attitudes on the web. As
a result, interest in automatically detecting sentences in which an opinion is expressed ([13] etc.),
the polarity of the expression ([14] etc.), targets([7] etc.), and opinion holders ([1] etc.) has been
receiving more attention in the research community. Applications include tracking response to
and opinions about commercial products, governmental policies, tracking blog entries for potential
political scandals and so on.
In the Sixth NTCIR Workshop, a new pilot task
for opinion analysis has been introduced. The pilot task has tracks in three languages: (Traditional) Chinese, English, and Japanese. In theSeventh NTCIR Workshop, the work on opinion analysis was continued with the Multilingual
Opinion Analysis Task, which has tracks in Chinese Simpliﬁed, Chinese Traditional, Japanese,and English. In this paper, we present an overview

of the NTCIR-7 MOAT test collection, task design, and evaluation results using the test collection across the Chinese, Japanese, and English
data.
This paper is organized as follows. In Section 2,
we explain the task design. Section 3, we brieﬂy
introduce the test collection used in the NTCIR7 Multilingual Opinion Analysis Task. Section 4
presents the annotation methodology. Section 5
details the evaluation methodology used, and explains the diﬀerences in the approaches taken with
examples. Section 6 presents evaluation resultsin (Traditional/Simpliﬁed) Chinese, Japanese andEnglish. Section 7 brieﬂy discusses the system approaches taken by the participants. Finally, we
present our conclusions in Section 8.

2

Task Design

2.1

Schedule

The time schedule for the NTCIR-7 Multilingual
Opinion Analysis Task is as follows.
Table 1: NTCIR-7 MOAT scheduleDate
2007-10-01
2008-08-01
2008-09-01
2008-09-10
2008-09-15
2008-09-26
2008-10-20∼27
2008-11-15
2008-12-16∼19

2.2

Event
First call for participation
Sample release (1 - 4 topics)
Formal run topic release
Japanese/Chinese (Traditional)
formal run results due
Chinese (Simpliﬁed)
formal run results due
English formal run results due
working paper submission
Camera-ready paper Submission
NTCIR-7 Meeting

Participants

Table 2 shows the number of participants per language and across languages. There are a total of
32 results submitted across all languages.

2.3

Task definition

In the NTCIR-7 MOAT, opinion annotation isextended to the sub-sentence level. Annotators
were instructed to annotate sentences in contiguous opinion expressions, with possibly multiple
opinion expressions per sentence. Two of the annotation features, whether a sentence is opinionated or not, and whether the sentence is relevantto the topic, are evaluated at the sentence level.

The other features, opinion holder, opinion target,and polarity, are evaluated at the opinion expression level.
Five evaluation subtasks
We set ﬁve subtasks in the evaluation, one ofwhich is mandatory, and the rest of which are optional. In Table 6, the mandatory subtask is todecide whether each sentence expresses an opinion or not. The optional subtasks are to decide
whether the sentences are relevant to the set topic
or not, to decide the polarity of the opinionatedsentences, to extract the opinion holder, and to
extract the opinion target.
1. Opinionated sentences
The opinionated sentences judgment is a binary decision for all sentences.
2. Relevant sentences
Each set contains documents that were foundto be relevant to a particular topic, such as
the one shown in Figure 1. For those participating in the relevance subtask evaluation,
each opinionated sentence should be judged
as either relevant (Y) or non-relevant (N)to the topic, and non-opinionated sentences
should be labeled Not Applicable (N/A). In
the NTCIR-6 Opinion Analysis Pilot task, allsentences were annotated for relevance, but in
the NTCIR-7 MOAT due to budgetary constraints, only opinionated sentences were annotated for relevance.
3. Opinion polarities
Polarity is determined for each opinion expression. In addition, the polarity is to be
determined with respect to the topic description if the sentence is relevant to the topic,
and based on the attitude of the opinion if
the sentence is not relevant to the topic. The
possible polarity values are positive (POS),
negative (NEG), or neutral (NEU.)
4. Opinion holders
Opinion holders are annotated for opinion expressions that express an opinion, however,
the opinion holder for an opinion expression
can occur anywhere in the document. The assessors performed a kind of co-reference resolution by marking the opinion holder for theopinion expression, if the opinion holder is ananaphoric reference noting the antecedent of
the anaphora. Each opinion expression may
have at least one opinion holder.
5. Opinion targets
Opinion targets were annotated in a similarTable 2: Number of participants
Language
Japanese English
Total
MultiJ-E-TC-SC
lingualJ-E-TC
Participants E-SC
E-J
TC-SC

manner to opinion holders. Each opinion expression may have at least one opinion target.
Sample (training) data
The NTCIR-6 Opinion Analysis Pilot Task
(Japanese, Traditional Chinese, and English) was
distributed as training data (approximately 32
topics per language.) Because the data is not thesame between the NTCIR-6 and NTCIR-7 tasks,
between 2 and 4 topics of sample data was distributed per language.
Evaluation metrics
Results for precision, recall, and F-measure will bepresented for opinion detection, and for sentence
relevance, polarity, opinion holders, and opinion
targets for those participants that elected to submit results for those optional portions. In Chinese,
Japanese, and English since all sentences were annotated by three assessors there is both a strict(all three assessors must have the same annotation) and a lenient standard (two of three assessors
have the same annotation) for evaluation, both ofwhich are being computed for all but the opinion
holder and target evaluation, which require somemanual judgment and will only be performed once
for each participating group. The formal deﬁnition is provided for the evaluation below.
1. Mandatory evaluation(a) Precision, Recall and F-measure of Opinion
using lenient gold standard.
(b) Precision, Recall and F-measure of Opinion
using strict gold standard.
2. Optional evaluation
For each optional subtask of the evaluation, polarity, relevance, opinion holders, and opinion
targets, the following information will be reported:(a) Precision, Recall and F-measure of Relevance using lenient gold standard.
(b) Precision, Recall and F-measure of Relevance using strict gold standard.

89ChineseTrad. Simp.
7
9

1
1
1

1

2
4

3

Test collection

3.1

Document sets

The test collection is based on the NTCIR-7
ACLIA Test Collection which includes newspaper documents from 1998 to 2001. The corpus
is a comparable corpus across the languages, with
topics shared across languages when enough documents exist in the corpus.
• It consists of Japanese data from 1998 to 2001
from the Mainichi newspapers.
• The Traditional Chinese data contains datafrom 1998 to 2001 from the China Times,
Commercial Times, China Times Express,
Central Daily News, China Daily News,
United Daily News, Economic Daily News,Min Sheng Daily, United Evening News, andStar News.
• The Simpliﬁed Chinese data contains documents from Xinhua News and Lianhe Zaobao
from 1998 to 2001.
• The English data also covers from 1998 to2001 with text from the Mainichi Daily News,
Korea Times, Xinhua News, Hong Kong
Standard, and the Straits Times.
The test collection was created using about
twenty queries from the NTCIR-7 ACLIA Task.
Relevant documents for each language weresearched for using baseline IR systems, and thenmanually assessed for relevance. Each topic contains from 5 to 20 relevant documents. Unlike the
NTCIR-6 Opinion Analysis Pilot Task, which has
fairly verbose topic descriptions, the topics in the
NTCIR-7 MOAT are short, and typically in theform of simple or complex questions.
As an example of the topics in the NTCIR-7
MOAT, please see Figure 1, which shows topic
M01, “Tell me about regenerative medicine”.
Table 3 shows the number of topics, the numberof documents, the number of sentences, and thenumber of opinion expressions for each language.

<TOPICDESCRIPTION><TOPIC>T01</TOPIC> <TITLE>Tell me about regenerative medicine.</TITLE> <NARRATIVE> I would like to know about regenerative
medicine, and how eﬀective it is. I would especially like to which diseases’ patients it
is eﬀective in treating. </NARRATIVE> </TOPICDESCRIPTION>

Figure 1: Topic title and narrative ﬁelds for topic N01
Table 3: Test collection size at NTCIR-7 MOAT
Language

T-Chinese
Japanese
English
S-Chinese

Topics
Sum
17
22
17
16

Sample3432Documents
Test
14
18
14
14

Sum
246
287
167
271

Sample58
38
25
19

The percentage of sentences that are opinionatedand relevant or that of opinion expressions about
polarities are also computed for both the strict
and lenient standards, as shown in Table 4.

3.2

TopicsTable 5 lists the titles of all the topics in the data
set. While only the English title is given, the topics and related meta-data as shown in Figure 1
have all been translated into each language.

4

Annotation

The NTCIR-6 Opinion Analysis Pilot Task extends previous work in opinion analysis [3, 8, 12] toa multilingual corpus. The initial task focuses on asimpliﬁed sentence-level binary opinionated or not
opinionated classiﬁcation as opposed to more complicated contextual formulations, but we feel thatstarting with a simpler task will allow for wider
participation from groups that may not have existing experience in opinion analysis.
In this task, we extended the annotation framework of polarities, opinion holders, and targets toa sub-sentence (opinion expression) level to improve the validness of the annotation if multiple opinion expressions are contained in one sentence. Table 6 summarizes the annotation subtasks, which are all being performed for all four
languages. All subtasks were annotated by three
annotators in each language: (Simpliﬁed or Traditional) Chinese, Japanese, and English. One
sample topic was used for inter-coder session to
improve the agreement between assessors.

4.1

Japanese annotation

The Japanese data was annotated by ﬁve annotators, and all topics were annotated by three assessors of them. They were given basic instructions

Sentences
Test
188
249
142
252

Sum
6,174
7,163
4,711
5,301

Sample
1,509
1,278
399
242

Test
4,655
5,885
4,312
4,877

Opinion Expressions(Sub-sentences/Clauses)
Sum
Sample
Test
N/A
N/A
4,657
7,569
1,348
6,221
4,733
404
4,329
7,523
570
6,953

about the annotation that was based on the same
strategy in NTCIR-6: i.e., the general or common knowledge and future plans were not counted
as opinions, and so forth. Then, they annotateda sample topic (M04) and held a meeting about
six hours afterwards to discuss discrepancies withthe explicit goal of trying to improve agreements
between annotators. The inter-annotator session
was limited to one topic, and for the remaining
twenty one topics, annotators worked independently. Even these independent results, for opinion and polarity annotation, the macro-averaged
value of the kappa coeﬃcient was above 0.70 and
0.60, which are considered high, as shown in Table 7.

Figure 2: Annotation tool at Japanese side

We developed a multilingual annotation tool to
output CSV and XML formats, but this tool wasused only at Japanese side because of time restriction. We also show the interface in Figure 2. With
the radio buttons left above, we can select values
such as opinionated or not in sentences or opinion
expressions (sub-sentences/clauses), relevant, positive, neutral, negative, and so on. If you would
like to annotate opinion holders or targets, you
can use the buttons left middle and you can dragLanguage

T-Chinese
Japanese
English
S-Chinese

Table 4: Opinion percentage in NTCIR-7 MOAT test collection
OpinionatedRelevant
Polarity(of Opinionated)(POS/NEG/NEU)
Lenient Strict Lenient Strict
Lenient
Strict
46.8
44.3
82.72
90.16 34.1 / 40.3 / 25.6 33.2 / 41.2 / 25.6
28.9
21.1
43.2
22.6
5.5 / 15.3 / 79.2
4.3 / 10.2 / 85.5
25.2
7.5
99.4
95.7
25.0 / 48.0 / 6.0
18.0 / 46.4 / 0.9
38.3
18.4
95.1
88.7
30.7 / 25.8 / 43.5 30.9 / 6.5 / 62.6

Table 5: NTCIR-7 OMAT topic titles
ID

Title
Japanese

M00
M01
M02
M03
M04
M05
M06
M07
M08
M09
M10
M11
M12
M13
M14
M15
M16
M17
M18
M19
M20
M21
M22
M23
M24

Microsoft Anti-monopoly
Regenerative medicine
American stance on depleted uranium bullets
The impact of 911 terrorist attacks on America’s economy
HIV-tainted blood scandal
Cosovo civil war
Incident with Nepal’s ruling family (royalty)
Attacks toward Chinese Indonesian people
Lawsuit American Government against Microsoft
Nuclear weapons tests
Suriyah in the Middle East Peace Process.
The relationship between AOL and NetscapeEl NinoThe relationship between China and Russia
Greenhouse gasses
The relationship between NATO and Poland
Thailand in the Asian economic crisis
Yasukuni Shrine
Chechin (Chechnia) civil war
Indonesian President Suharto
Nuclear missile abandonment of North Korea
Airplane crashes in Asia
The ﬂoods in the Mainland China
The births of the cloned animals known to the world
The responses of other countries to Lockerbie Air Disaster

N00
N01
N02
N03
N04
N05
N06
N07
N08
N09
N10
N11
N12
N13
N14
N15
N16
N17
N18
N19
N20
N21

Language ID
EnglishChinese
Traditional Simpliﬁed
N01
N02
N03
N04
N05
N06
N07
N08
N09
N10
N11
N12
N13
N14
N15
N16
T01

Table 6: Five annotation subtasks at NTCIR-7 Multilingual Opinion Analysis Task
Subtasks
ValuesReq’d?
Annotation Unit
Opinionated Sentences
YES, NO
Yes
Sentence
Relevant Sentences
YES, NONo
Sentence
Opinionated Polarities POS, NEG, NEU
No
Opinion Expression
String, multiple
No
Opinion Expression
Opinion Holders
Opinion Targets
String, multiple
No
Opinion Expression

N02
N03
N05
N07
N09
N10
N11
N12
N14
N15
N16

N01
N02
N03
N05
N06
N07
N08
N09
N10
N11
N12
N13
N14
N15
N16

T96
N04
N13
N08
N01
N06
N04

Table 7: Annotators agreements by topics at Japanese side
TopicID

M00
M01
M02
M03
M04
M05
M06
M07
M08
M09
M10
M11
M12
M13
M14
M15
M16
M17
M18
M19
M20
M21

Assessor
1
2
3
A B C
A B C
A B C
A B C
A B C
A B C
A B C
A B C
A B C
A B C
A B C
A B C
E B C
A B C
E B C
A B C
A B C
A E D
A B D
A B D
A B C
A B D
Macro Avg
Micro Avg

Sample/Test
SampleSampleSample
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Sample
Test
Test
Test
Test

Opinionatedness
a1-a2
a1-a3
a2-a3
0.8029 0.7465 0.6619
0.8707 0.8742 0.7511
0.7931 0.8586 0.7416
0.7368 0.7276 0.6652
0.7837 0.8234 0.8202
0.6604 0.5969 0.4852
0.6542 0.6532 0.5932
0.7369 0.7143 0.6772
0.7056 0.6098 0.6773
0.7009 0.6238 0.6658
0.6663 0.7025 0.6747
0.6728 0.5875 0.6641
0.9296 0.8219 0.7896
0.6392 0.6874 0.6092
0.8314 0.8169 0.8212
0.6700 0.5707 0.7190
0.7457 0.6375 0.6782
0.7711 0.7965 0.8504
0.7278 0.5422 0.6242
0.6477 0.6275 0.6790
0.7228 0.7798 0.8200
0.7025 0.6430 0.8058
0.71350.7128opinion holder elements in the right pane. Whenyou split the sentence into multiple opinion expressions, you can push the button left below.

4.2

Traditional Chinese annotation

In NTCIR-6, we deﬁned that one sentence should
end with a period, so that we could segment
sentences automatically. However, we found thelength of many sentences was extremely long, andthese sentences were very complex in annotating
their opinionated features. Therefore, in NTCIR7, we segmented sentences into their opinion units
as possible as we could before annotations. In theend, we only found two sentences with more than
one opinion clauses in testing data. Therefore, we
evaluated results at sentence level since only two
extra opinion clauses were found.
After the sentence segmentation, a pool of ten
annotators were used to annotate the documents,
with three annotators per topic. Prior to annotation, the annotators underwent a two-hour-longorientation period where the purpose of the annotation was explained, and examples of sentences
and their annotations were given. After this orientation session, the annotators started their practice session for three hours. They were free to ask
the annotation coordinator questions about speciﬁc sentences if they were unsure of the label-

Kappa Coeﬃcient
Polarity
a1-a2
a1-a3
a2-a3
0.6996 0.5777 0.5476
0.7889 0.7448 0.6817
0.6355 0.6330 0.6544
0.6812 0.6157 0.5390
0.6605 0.7327 0.6993
0.6350 0.6266 0.4443
0.6375 0.6339 0.6604
0.7113 0.6688 0.6276
0.6040 0.4648 0.5104
0.6700 0.5419 0.5815
0.6389 0.6625 0.6355
0.5859 0.4890 0.6421
0.8969 0.7488 0.7177
0.4987 0.5334 0.5538
0.7708 0.7235 0.7634
0.6423 0.4812 0.5999
0.7137 0.5333 0.5785
0.6194 0.6380 0.6693
0.6587 0.4834 0.5279
0.6164 0.5532 0.5388
0.6902 0.7415 0.7669
0.7088 0.6259 0.6924
0.6341
0.6380a1-a2
0.54320.77950.64400.66130.71710.61940.56990.55020.64570.60160.54400.68170.90290.60520.68990.63380.34020.51080.70750.50350.38430.6719Relevance
a1-a3
a2-a3
0.3959 0.6100
0.8080 0.6805
0.7177 0.4971
0.6695 0.6236
0.7396 0.7366
0.5394 0.4747
0.6356 0.5568
0.5887 0.4611
0.6171 0.6233
0.5209 0.5600
0.7033 0.5537
0.5666 0.6331
0.7053 0.6777
0.5843 0.5196
0.5848 0.4645
0.2903 0.3027
0.6054 0.2705
0.4580 0.7129
0.5744 0.6335
0.4337 0.6054
0.4525 0.7346
0.5947 0.7471
0.59050.5785ing, and they can discuss with each other in thissession to ensure consistency between the annotators in those cases. After the practice session they
started to annotate oﬃcially, and they were notallowed to discuss their labeling with each otherin this phase. They could still ask the annotation
coordinator if they had any questions.
For annotations of the traditional Chinese corpus, two stages were performed. The ﬁrst stage,
as mentioned in the last paragraph, we had three
annotators annotate each topic. In the second
stage, we calculated the pairwise kappa of opinionated tags among them. If any kappa below 0.3
appeared, we asked the forth annotators to annotate this topic. And if any kappa below 0.3 still
appeared among three sets of data with highest
kappa values, we had the ﬁfth annotator generate
new data again. Finally, we selected three setsof data with highest pairwise kappa values as theﬁnal data to generate the gold standard.
For each topic the agreement between the three
annotators was computed, and the macro average (over topics) and the micro average (over sentences) are shown in Table 8. Because the corpus is annotated by ten annotators, the column
annotator n (n=1,2,3) does not necessary denotethe same person. Therefore, the values of macroand micro average are calculated by the average ofthree annotators. Though agreements vary from

topics, we still can see worse agreements for the
relevance tags from statistics. Note that we reannotated according to the pairwise kappa valuesof opinionated tags, and it may be one of the reason that we had higher agreements among opinionated and polarity tags. We also found that thereason why we had low relevance agreements is
because some annotators gave many more Y relevance tags than the others instead of real contradictions. In other words, annotators might haveproblems in judging which sentences are relevantby only reading topic descriptions. This might
be due to the shorter descriptions of topics weadopted from the ACLIA cluster, which contains
multilingual QA tasks.

4.3

Simplified Chinese annotation

The simpliﬁed Chinese data was annotated by
twelve annotators, and all topics were annotatedby three of them. Prior to formal annotation,
they were given a basic instruction note about the
purpose of the annotation and annotation rules
based on English 2008 Opinion Annotator Sample Instructions by David and Lun-Wei Ku’s OAT(Opinion Annotation Tool) 3.0 Manual. We use
this tool for our annotation work. Firstly, fourannotators are selected to do the sample annotation. They annotated two sample topics and helda meeting about 4 hours to discuss the discrepancies and the ambiguous statements in the instruction rules. Then all the annotators held anothermeeting about 4 hours to learn how to do the annotation work based on the ﬁnished sample date
and the sample in OTA3.0 manual. Four teams
are set up and the person who done the sample
annotation are set as the leader of the four-person
team. The annotators can ask any questions to
the team leader about the speciﬁc sentences ifthey were unsure of the labeling. Three annotators will annotate each sentence in the data set for
opinion features. When more than one opinion ispresent in a sentence, the annotators will separate
the sentence into separate opinion clauses, which
are then annotated for the opinion features. There
are also sentence-level features that only pertain
to the sentence as a whole.
After all annotation data ﬁnished, we checkthem by calculating the pairwise kappa for relevance and polarity tags. If the kappa value of
one topic is obviously below than others, we ask
the team leader to check this topic again in caseany misunderstanding by one of the annotator in
his team. Finally, we calculated the agreement
between the three annotators, and the macro average (over topics) and the micro average (oversentences) are shown in Table 9. Because the cor-pus is annotated by twelve annotators, the column
annotator a1, a2, a3 does not necessary denote thesame person.

4.4

English annotation

The English data was annotated using a pool of
six annotators. The annotators underwent an initial two hour training session, then annotated a
training topic over the span of a week. The annotators then took part in a second two hour meeting where we discussed examples from the training topic that were contentious, and strategies for
improving consistency between annotators. Afterthe two training meetings, annotators began workon the sample data and formal run data. The
annotators used the same tool that was used for
the Sipliﬁed and Traditional Chinese annotation,
Lun-Wei Ku’s Opinion Annotation Tool version
3.0.
After the annotation for the topics was completed, for topics with low kappa agreement the
annotators were asked to review their annotationand conﬁrm their annotation. The kappa results
are shown in Table 10.

5Evaluation Approach

5.1

Traditional Chinese evaluation

5.1.1

Opinionated / relevance

Relevance, opinion extraction, polarity detection,holder and target identiﬁcations were evaluated at
traditional Chinese side. Among them, relevanceand opinion extraction adopted the same metric,
while metrics for polarity detection, holder andtarget identiﬁcation are similar.
Under the strict evaluation, all three annotators
must agree on the classiﬁcation of the sentence tobe counted as either an opinionated or relevant
sentence. Under the lenient evaluation, two of thethree annotators must agree on the classiﬁcationof the sentence for it to be counted. Precision (P)is computed as#system correct.
#system proposed

Recall (R) is computed as#system correct.
#gold

And the F-measure (F) is computed as
2×P×R
.
P+R

where the number of sentences is either the number of opinionated or relevant sentences according
to the strict or lenient criteria.

Table 8: Annotators agreements by topics at traditional Chinese side
TopicID(TrC/Multi-lingual)
N03/M03
N04/M19
N05/M05
N06/M23
N07/M07
N08/M21
N09/M09
N10/M10
N11/M11
N12/M12
N13/M20
N14/M14
N15/M15
N16/M16

Assessor
1
2
3
A B C
A B C
A B C
A B C
A B C
A B C
A B C
A B C
A B C
E B C
A B C
E B C
A B C
A B C
Macro Avg
Micro Avg

Sample
/Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test

Opinionatedness
a1-a2
a1-a3
a2-a3
0.4958 0.7276 0.4968
0.3120 0.2331 0.5824
0.4145 0.4796 0.4053
0.3429 0.3742 0.7005
0.4176 0.5129 0.4520
0.3880 0.3760 0.5200
0.3307 0.5168 0.3535
0.6204 0.5814 0.5863
0.2915 0.3863 0.5762
0.5382 0.4147 0.4176
0.4994 0.3710 0.3624
0.4580 0.5004 0.4885
0.4178 0.5343 0.5837
0.3926 0.4873 0.2987
0.45810.4706

Kappa Coeﬃcient
Polarity
a1-a2
a1-a3
a2-a3
0.7162 0.7830 0.5339
0.5657 0.6360 0.8691
0.8207 0.7408 0.8530
0.8042 0.9506 0.8504
0.5499 0.5927 0.5797
0.8825 0.8446 0.9674
0.7772 0.7338 0.8307
0.7500 0.8238 0.7439
0.8880 0.9566 0.9109
0.9603 0.6497 0.6551
0.3624 0.7635 0.8312
0.7912 0.9330 0.7313
0.8577 0.7936 0.7427
0.7818 0.7531 0.8168
0.77090.7699a1-a2
0.52280.09300.21700.32250.58230.56450.20270.2496
0.09920.30180.36240.34930.07230.0098

Relevance
a1-a3
a2-a3
0.1810 0.2648
0.2583 0.1800
0.1708 0.0489
0.4890 0.5028
0.7100 0.7221
0.6005 0.6960
0.4923 0.4162
0.2355 0.5224
0.2707 0.3729
0.1626 0.2069
0.3624 0.6799
0.0908 0.3497
0.0313 0.5572
0.3973 0.0595
0.33290.3376

Table 9: Annotators agreements by topics at Simpliﬁed Chinese side
TopicID(SiC/Multi-lingual)
N03/M03
N04/M24
N05/M05
N06/M06
N07/M07
N08/M08
N09/M09
N10/M10
N11/M11
N12/M12
N13/M13
N14/M14
N15/M15
N16/M16

Assessor
1
2
3
A B C
A B C
A B C
A B C
A B C
A B C
A B C
A B C
A B C
A B C
A B C
A B C
A B C
A B C
Macro Avg
Micro Avg

Sample
/Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test

Opinionatedness
a1-a2
a1-a3
a2-a3
0.3862 0.3890 0.5455
0.3975 0.4567 0.9222
0.5209 0.4295 0.7442
0.3847 0.3392 0.8925
0.6045 0.5050 0.6577
0.4286 0.4592 0.6936
0.2518 0.3163 0.4856
0.4913 0.4736 0.5072
0.6244 0.6148 0.4436
0.2906 0.1628 0.1058
0.1297 0.3718 0.1738
0.3711 0.4435 0.4932
0.2173 0.5064 0.3147
0.1139 0.4753 0.1835
0.43620.3862

Kappa Coeﬃcient
Polarity
a1-a2
a1-a3
a2-a3
0.2937 0.3119 0.5577
0.3067 0.3225 0.7603
0.3670 0.3932 0.5682
0.1890 0.1935 0.8081
0.3782 0.4166 0.4433
0.3151 0.4551 0.5562
0.3119 0.3864 0.4649
0.3798 0.4593 0.4382
0.5119 0.4985 0.3908
0.1997 0.1748 0.0952
0.1044 0.3369 0.1791
0.3283 0.3936 0.4213
0.2089 0.4390 0.2602
0.1197 0.3733 0.1497
0.36340.3314a1-a2
0.7786
0.39261.00000.49560.69460.52030.50930.39580.78980.12610.32430.61570.93470.9316

Relevance
a1-a3
a2-a3
0.7368 0.8710
0.3926 1.0000
1.0000 1.0000
0.6647 0.6647
0.0534 0.0139
0.6299 0.7668
0.5879 0.6501
0.0401 0.1773
1.0000 0.7898
0.3130 0.3225
0.6422 0.1898
0.6157 0.5272
0.9321 0.9538
1.0000 0.9316
0.61850.6030Table 10: Annotators agreements by topics for English data
TopicID

N03
N04
N05
N06
N07
N08
N09
N10
N11
N12
N13
N14
N15
N16

5.1.2

Assessor
1
2
3
A B C
A B C
A B C
A B C
A B C
D E F
A B C
D E F
A B C
D E F
D E F
D E F
A B C
D E F
Macro Avg
Micro Avg

Sample
/Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test

Opinionatedness
a1-a2
a1-a3
a2-a3
0.36750.3066
0.34300.38500.29400.38500.24830.26370.22030.11320.11890.08680.17820.21010.17840.3846
0.37900.37040.42180.4136
0.40810.45550.42920.44050.25710.28390.2897-0.0101 -0.0080 -0.0101
0.30470.2996
0.28380.05490.0171
0.05490.3516
0.41170.35160.07060.0831
0.05250.23690.2343Kappa Coeﬃcient
Polarity
a1-a2
a1-a3
a2-a3
0.39440.34050.35260.31080.22260.31080.25810.18680.23400.05170.07350.03720.12910.12050.12190.45620.41930.44090.16410.13490.16800.38370.39790.38370.00980.01770.0114-0.0111 -0.0065 -0.0111
0.37250.30040.35430.12420.12120.12930.12510.19380.12510.08600.08200.08340.19580.2156

5.1.3

Opinion holder/targets

Polarity

The LWK approach gives two evaluation results.
The traditional metric, which is the same as thatused in NTCIR6, evaluates opinionated sentences
correctly reported by participants’ systems under
the opinionated strict or lenient gold standards.
Under the traditional metric, a Set Precision (SP) is computed as#system correct(polar = POS, NEU, NEG).
#system correct(opn = Y)

The other recall based metric evaluates all opinionated sentences under the opinionated strict orlenient gold standards.
Under this recall based metric, Precision iscomputed as#system correct(polar = POS, NEU, NEG).
#system proposed(opn = Y)

Recall is computed as#system correct(polar = POS, NEU, NEG).
#gold(opn = Y)

And the F-measure is computed as
2×P×R
.
P+R

The polarity for the sentence is the polarity
with the largest number of votes by the annotators. In cases where the polarity of the sentenceis ambiguous, POS + NEU the gold standard is
POS, for NEG + NEU the gold standard is NEG,
for POS + NEG the gold standard is NEU, and for
POS + NEU + NEG the gold standard is NEU.

a1-a2
0.56840.56650.44690.42150.24270.35340.29130.52970.11670.12070.40820.34570.41880.1331

Relevance
a1-a3
a2-a3
0.5017 0.5561
0.4556 0.5665
0.3951 0.4469
0.3491 0.4317
0.2353 0.2470
0.3577 0.3314
0.2410 0.2913
0.4641 0.5186
0.0970 0.1206
0.0934 0.1207
0.3353 0.4034
0.5894 0.3085
0.4276 0.4188
0.1247 0.1346
0.34590.3282The evaluation at traditional Chinese for opinion
holders / targets is semi-automatic. All possible
aliases of each opinion holder are generated manually ﬁrst, for example, the names of holders withor without their titles. The results then are evaluated according to this information by keyword
matching. To ensure the correctness of the evaluation, every record which is diﬀerent from all aliasesof the correct holders is checked manually again. If
any correct answer is found, it is added to the gold
standard. At last, all runs are evaluated according to the ﬁnal gold standard by keyword matching and their precision, recall and F-measure arecalculated.
The traditional metric (Holder-T, Target-T)
and the recall based metric (Holder-RB, TargetRB) are both adopted for evaluation again. ¡ Theyevaluate opinionated sentences correctly reported
by participants’ systems under the opinionatedstrict or lenient gold standards. In these opinion
sentences, all holders/targets annotated by three
annotators are viewed as correct answers. Participants can report anyone of them to get the score.
Under this traditional metric, Precision is computed as#system correct(polar = POS, NEU, NEG).
#system correct(opn = Y)

Recall is computed as#system correct(polar = POS, NEU, NEG).
#system correct(opn = Y)

And the F-measure is computed as
2×P×R
.
P+R

Under the recall based metric, Precision is computed as

not agree on the relevance of the sentence. NA isused when a sentence is not opinioated. Precision
and Recall are computed as given below.

#system correct(polar = POS, NEU, NEG).
#system proposed(opn = Y)Precision(P)=Recall(R)=

Recall is computed as#system correct(polar = POS, NEU, NEG).
#gold(opn = Y)

And the F-measure is computed as
2×P×R
.
P+R

Notice that if we are not sure the proposed answer is the same entity as the correct answer, it istreated as a wrong answer. For example, if the correct holder is ”the president of America” but theparticipant reports ”the president”, there will not
be a match. And also the proposed answer without the resolution of the anaphor or the coreference is treated as correct. Of course, the resolved
form of an anaphor or a coreference is correct.

5.2

Simplified Chinese evaluation

The evaluation approach we used is almost thesame as in the traditional Chinese side. However,when calculating the value of #system-correct
holder or #system-correct target, we use the automatically matching method for tight schedule.
In the lenient metric, for each sentence the holders/targets of the three annotators tagged will be
combined, if the system proposed holder/target
can match one of them, it will be considered correct. In the strict metric, the holder/target of thethree annotators must be identical, only when thesystem proposed holder/target can match the annotators’ holder/target, it will be considered correct. This is one of the reasons that the holder andtarget performances are relative lower than otherlanguages.

5.3

English evaluation

5.3.1

Opinionated sentence

The English evaluation takes the same lenient andstrict deﬁnitions as the other languages, where twoannotators must agree for the lenient case, and all
three for the strict case. Opinionated sentence
precision, recall, and F-measure is deﬁned as in
Section 5.4.1.
5.3.2

Relevance

For relevance evaluation, a contingency table forthe categories YES, NO, NA, and NONE, where
NONE is the category used when annotators do#system correct(rel = YES|NO|NA).
#system proposed(rel = YES|NO|NA)#system correct(rel = YES).
#assessors agreeed(rel = YES)

Unfortunately, for English while relevance wasdeﬁned only to be annotated for Opinionatedsentences, many annotators also annotated nonopinionated sentences, so the denominator for recall is larger than the set of all opinionated sentences.
5.3.3

PolarityIn the NTCIR-6 evaluation polarity was weighted
according to the annotator scores, but this year
the English evaluation is more similar to the Chinese and Japanese evaluations: two or more annotators must agree in the lenient case for the annotation to be used, or all three annotators for thestrict evaluation. The evaluation script creates acontingency table for the categories POS, NEU,
NEG, and NONE, where NONE is category thatis used when a sentence is not opinionated or notenough annotators agree on the polarity.
Precision(P)=Recall(R)=F measure(F)

=

5.3.4

#system correct(pol = POS|NEG|NEU).
#system proposed(pol = POS|NEG|NEU)#system correct(pol = POS|NEG|NEU).
#assessors agreeed(pol = POS|NEG|NEU)
2×P×R
.
P+R

Opinion Holder and Target

Opinion Holder and Target evaluation under the
DKE strategy used a perl script to implement asemi-automatic evaluation. For each document,
an equivalence class is created for each opinion
holder or target, and system opinion holders or
targets for a given sentence are matched using exact string matches to the opinion holders or targets in the equivalence class. Exact matches arecounted as correct, and if no matches are foundthen a human judge1 is asked to determine if thesystem answer matches ones of the opinion holders
or targets in the equivalence class for the sentence.
If there is match, the system opinion holder or target is added to the equivalence class, otherwise it
is marked as a known incorrect opinion holder ortarget.
The initial database of opinion holder and target equivalence classes is created by adding the
1 For this evaluation, the co-author David Kirk Evansopinion holders and targets marked by the annotators. The database grows with each evaluatedsystem, and after the ﬁrst run for each system
subsequent runs can be done automatically using
the opinion holder and target database to match
opinion holders.
Precision is computed as the number of correctly matched opinion holders or targets divided
by the number of oﬀered opinion holders or targets. The denominator for Recall is computed byassuming one opinion holder and one target foreach opinion unit.

5.4

Japanese evaluation

5.4.1

Opinionated sentence

For opinionated sentence evaluation, we took thesame approach with other languages. For lenient
standards, we prepared the correct answer set ofsentences that two of three assessors agreed themas opinionated. For strict standards, we prepared
the answer set of sentences that three of three assessors agreed them as opinionated. Then, theprecision, recall, and F-measure are deﬁned as follows:
Precision(P)=Recall(R)=F measure(F)

=

5.4.2

#system correct(opn = Y).
#assessors agreeed(opn = Y)
2×P×R
.
P+R



system correct(rel = Y )&system proposed(opn = Y )#system proposed(rel = Y )&system proposed(opn = Y )&assessors agreed(opn = Y )#F measure(F)=system correct(rel = Y )&system proposed(opn = Y )#assessors agreed(rel = Y)Polarity

For polarity judgment, we also took the same approach with the relevance judgment, based on thesame reason. In this subtask, however, the evaluation was conducted at the opinion expression (subsentence/clause) level, not at the sentence level.



#
=

=

The participants can also evaluate the results
for whole sentences by using the evaluation script
we provided with the option ‘-w’. In this case, thedenominator of the precision is counted on all sentences, and three metrics are computed based on
the same approach with the opinionated sentence
judgment case.

5.4.3

For relevance judgment, we only annotated relevant information for opinionated sentences. Then,the precision, recall, and F-measure are deﬁned
based on opinionated sentences as follows.

Recall(R)• The denominator of the precision is counted
on the opinionated sentences that the assessors agreed. This corresponds to the set precision metric at the other language sides. However, with the pure precision, the participants
who submitted the results for all sentences
will be disadvantage, because their correct
answers in the non-opinionated sentence are
not counted with the former reason. Basedon this discussion, we decided to deﬁne set
precision as the precision metric at Japaneseside.

#system correct(opn = Y).
#system proposed(opn = Y)

Relevance

Precision(P)• The numerators of the precision and recall
was counted on the opinionated sentences thesystem proposed, to exclude N/A cases, asdeﬁned in Section 2.3.

.


.

2×P×R
.
P+R

The reason that the precision at Japanese side
was computed as above is to treat the diﬀerent
number of submitted sentences from all the participants equally: some participants submitted theresults for all sentences, while the other participants submitted the results only for opinionatedsentences.

For the agreement estimation between assessors, we implemented YS method (= to use in
the answer set of polarity as POS/NEG/NEU setsthat the assessors agreed strictly or leniently) and
DKE method (= in strict case, the same approachwith YS; in lenient case, weight the answer set according to the number of agreed assessors) in theevaluation script, that were deﬁned in NTCIR-6
workshop [9]. We set YS method as a default function and DKE method as an optional function. We
also implemented LWK method (= majority voting) as another optional function, that was explained in Section 5.1.2.

5.4.4

Opinion holder & target

Unfortunately, we did not receive the submissionsof opinion holder/target subtasks from the participants at Japanese side (except the organizer) this
time.

6

Evaluation Results

6.1

Simplified Chinese

Table 11 shows the evaluation results of opinionated, relevance, and polarity based on the lenientand strict gold standards. Table 12 shows the evaluation results of holder and target based on thelenient and strict standards we described in 5.2.

6.2

EnglishTable 13 lists the evaluation results of the opinionated, relevance, and polarity analysis for English
based on lenient and strict standards.

6.3

Japanese

Table 15 lists the evaluation results of opinionated,
relevance, and polarity analysis at Japanese side
based on lenient and strict standards.

6.4

Traditional Chinese

Table 16 lists the evaluation results for relevance,
opinion extraction and polarity detection based onlenient and strict gold standards. The results ofrelevance, opinion extraction and polarity detection are evaluated automatically while holder andtarget identiﬁcations manually, so they are separated evaluated and listed in two tables.
For evaluation of opinion holder and target
identiﬁcations, we applied both the sentencebased traditional evaluation (Holder-T) and therecall-based evaluation (Holder-RB), as shown in
Table 17. In both evaluation metrics, precision,
recall and F-measure are listed, but their deﬁnitions, introduced in LWK evaluation approach,are diﬀerent. Because the denominators in theformulae for calculating the precision and recall
in the traditional evaluation are the same, the values of precision, recall and f-measure are the same.
Note that the traditional metric may have preference in a high precision opinion system, while theperformance of the opinion extraction task may
also inﬂuence the performance evaluated by therecall-based metric.

7

Discussions on Participant
System7.1

EnglishNine groups submitted runs for the English evaluation.

The Sussex group use very few languagedependent features, and have an interesting segmentation method for Japanese and Chinese.
Their relevance system is unsupervised and compares word ranks across topics to make a relevancedecision, while they used the training data to collect a list of opinionated terms, then manually selected a small number of terms and an automatically expanded set of related terms as features fortheir opinionated judgement system.
The North Eastern University group’s submission for English took advantage of an existing sentiment lexicon with a rule-based system and compared that to a Naive Bayes system trained over
the NTCIR-6 Opinion Pilot Task data and MPQA
data.
The TUT submission separates opinions into
two classes: those from the author’s point of view,and those from an authority’s point of view. Theyuse a χ2 test to identify syntactic and semantic
features from the NTCIR-6 and MPQA corpora,and compare SVM voting and Multi-label classiﬁcation approaches. They participated in the opinionated, relevance judgement, polarity classiﬁcation, and opinion holder identiﬁcation sub-tasks.
The KLE group uses data from SentiWordNet
and Levine’s Verb Classes to identify opinionatedfeatures, and weight them using a BM25 algorithm and the NTCIR-6 corpus to estimate somesmoothing values for the opinionated component.
They determine polarity based on the weights ofthe features from the sentiment lexicons, and theyuse features from a dependency parse to determine
the opinion holder.
The University of Neuchâel group takes alargely language-independent approach to thetask, and use a statistical method to identify tokens (in the English case words, but unigrams andbigrams in Japanese and Chinese) that are usefulfor opinionated and polarity detection.
The MIRACLE team use a machine learning
approach for the English and Japanese opinionated, polarity, and relevance tasks. They use seedterms from an existing lexicon that have been annotated for polarity as features and a KNN clustering approach for opinionated and polarity classiﬁcation. Relevance uses a distance metric between expanded semantic space vectors for eachof the topics and the sentences, and data from
NTCIR-6 for parameter optimization.
The UKP group use a sequential SVM to labelwords, and compute the opinion holder, opinionated, and polarity tasks simultaneously over thelearned tags. They use both lexical and syntactic features and existing sentiment lexicons, andinvestigate domain adaptation using a structural
correspondence learning approach.

Table 11: Simpliﬁed Chinese opinionated/relevance/polarity analysis results
Group

RunID

BUPT
ICLPKUICLPKUNEUNLP
NLCL
NLCL
NLCL
NLPR
NLPR
NLPR
NLPR
NTU
NTU
NTU
TTRD
TTRD
WIA ∗∗ISCAS∗BUPTICLPKUICLPKUNEUNLP
NLCL
NLCL
NLCL
NLPR
NLPR
NLPR
NLPR
NTU
NTU
NTU
TTRD
TTRD
WIA ∗∗ISCAS∗
∗
∗∗

=
=

L/S

1
1
2
1
1
2
3
1
2
3
4
1
2
3
1
2
1
1
1
1
2
1
1
2
3
1
2
3
4
1
2
3
1
2
1
1

L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S

P
0.604
0.48030.44870.47210.44250.48220.4316
0.58220.588
0.45510.57690.59390.59560.59560.412
0.4456
0.5862
0.46490.63120.4486
0.39840.43580.38570.44250.36670.6096
0.61290.41970.59730.6314
0.63430.63430.34810.39580.60980.4271

OpinionatedRF
0.39910.48070.80040.60030.79830.57450.7116
0.5676
0.39910.41970.3686
0.41780.69880.53360.77530.665
0.48420.53110.57250.50710.56390.57030.60890.60130.60670.60110.60670.6011
0.9636
0.57720.756
0.56070.82080.68390.74420.57230.44210.52
0.82070.58010.82520.53730.73390.54690.402
0.39370.38980.41440.706
0.48270.892
0.724
0.55010.57980.637
0.506
0.64590.62070.75170.68630.74940.68710.74940.6871
0.96990.51240.755
0.51930.89640.72590.81180.5597

P
0.97750.97750.963
0.9752
0.97140.9656
0.9796
0.97670.95070.968
0.994
0.97030.98450.98450.9736
0.98480.98270.97480.98780.9866
0.96310.97590.99690.9828

RelevanceR
N/A
0.65590.6559N/A
0.32580.2799
0.585
N/A
N/A
N/A
N/A
0.76930.57980.5796
0.69810.73630.50320.9288N/A
0.67430.6743
N/A
0.33260.2846
0.5897
N/A
N/A
N/A
N/A
0.78590.59690.59430.70060.74870.524
0.9369F

Polarity
S-P

0.785
0.785

0.45050.45240.48690.43490.73020.85640.72840.72750.80510.8364
0.6682
0.94910.4956
0.50790.51590.43480.49470.74190.80040.80040.28360.28070.49590.44150.73710.87020.74410.74180.81120.84740.687
0.95930.33780.3611
0.373
0.2882
0.39230.5329Recall-based Polarity
P
R
F
N/A
0.21640.36060.27050.203
0.36120.2599
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
0.29440.30180.298
0.30250.30820.30530.30720.313
0.31010.17910.41890.251
0.22040.374
0.27740.43480.60890.5074
N/A
N/A
0.12720.23270.16450.11180.23160.1508
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
0.21330.25390.23180.229
0.2706
0.2481
0.2366
0.27950.25630.10030.27950.1476
0.15530.29620.20380.32500.47770.3868
N/A

organizer
late submission

Table 12: Simpliﬁed Chinese opinion holder/target analysis results
Group
ICLPKUICLPKU
TTRD
TTRD
NLPR
NLPR
NLPR
NLPR
NTU
NTU
NTU
WIA∗∗
ICLPKUICLPKU
TTRD
TTRD
NLPR
NLPR
NLPR
NLPR
NTU
NTU
NTU
WIA∗∗
∗∗

=

Run
ID
1
2
1
2
1
2
3
4
1
2
3
1
1
2
1
2
1
2
3
4
1
2
3
1

L/S
/S
L
L
L
L
L
L
L
L
L
L
L
L
S
S
S
S
S
S
S
S
S
S
S
S

late submission

P
0.41240.40950.11290.12700.42860.44970.40370.42980.29090.03970.15870.66560.41040.42750.17190.11250.47590.4821
0.46890.47150.38390.02680.16520.7574

Holder-TR
0.41240.40950.11290.12700.42860.44970.40370.42980.29090.03970.15870.66560.41040.42750.17190.11250.47590.4821
0.46890.47150.38390.02680.16520.7574

F
0.41240.40950.11290.12700.42860.44970.40370.42980.29090.03970.15870.66560.41040.42750.17190.11250.47590.4821
0.46890.47150.38390.02680.16520.7574

P
0.19810.18380.04640.0566
0.24950.26450.18380.24790.17280.0236
0.09450.3901
0.09370.08290.02820.02180.17190.16880.09800.15580.13920.00980.06050.2817

Holder-RB
RF
0.33010.2476
0.32690.23530.1081
0.06490.09580.0711
0.33230.28500.21780.23890.23110.20470.24240.24510.17710.17490.02410.02390.09630.09540.54630.45520.32160.14510.33630.13290.16080.04800.08190.03450.40350.24100.23680.19710.28660.14610.26610.19650.25150.17920.01750.0126
0.10820.0776
0.67540.3976

P
0.00330.00340.0151
0.05460.45050000.06250.5185

Target-TR
0.00330.00340.0151
0.0546
N/A
N/A
N/A
N/A
N/A
N/A
N/A
0.45050000.0625
N/A
N/A
N/A
N/A
N/A
N/A
N/A
0.5185F
0.00330.00340.0151
0.05460.45050000.06250.5185

Target-RB
R
0.00270.00270.01440.0412
N/A
N/A
N/A
N/A
N/A
N/A
N/A
0.26400.36970000000.00610.0479
N/A
N/A
N/A
N/A
N/A
N/A
N/A
0.10770.4795P
0.00160.00150.00620.0243F
0.00200.00190.00870.03060.30810000.01080.1759

Table 13: English opinionated/relevance/polarity analysis results
Group
ICU
ICU
kle
kle
kle
MIRACLENEUNLPNEUNLP
NLCL
NLCL
NLCL
sics
sics
sics
TUT
TUT
TUT
UKP07
UKP07
UniNe
UniNe
UniNe
ICU
ICU
kle
kle
kle
MIRACLENEUNLPNEUNLP
NLCL
NLCL
NLCL
sics
sics
sics
TUT
TUT
TUT
UKP07
UKP07
UniNe
UniNe
UniNe

RunID
1
2
1
2
3
1
1
2
1
2
3
1
2
3
1
2
3
1
2
1
2
3
1
2
1
2
3
1
1
2
1
2
3
1
2
3
1
2
3
1
2
1
2
3

L/S
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S

P
0.24350.24350.35290.37510.27360.59520.35220.29520.37800.33840.2766
0.41920.2847
0.2916
0.31850.32820.31850.33050.33050.33220.37740.38290.07430.07430.11090.11950.08080.28570.11050.08810.11680.10890.08380.13360.08320.08790.09610.10390.09610.10260.10260.10500.11960.1225

OpinionatedRF
0.36870.29330.36870.29330.72720.47520.54100.44300.93270.42310.01160.02270.77880.48510.89860.44440.10140.15990.18620.24020.84610.41690.61010.49700.84520.42590.92350.44330.40920.35820.25620.28780.40920.35820.90600.48440.90600.48440.69950.45040.57600.45600.55300.45250.37770.12410.37770.12410.76780.19380.57890.19810.92570.14870.01160.02220.82040.19470.90090.16050.10530.11070.20120.14130.86070.15270.65330.22190.82970.15120.93500.16070.41490.1561
0.27240.15040.41490.1561
0.94430.18500.94430.18500.74300.18400.61300.20010.59440.2032P
0.27580.27570.37410.12960.1746
0.48170.20920.16470.20920.41700.34230.33050.0981
0.09810.08530.05260.07440.16440.07400.06150.07400.16140.14310.1382

RelevanceR
0.36480.3648N/A
N/A
N/A
0.3189N/A
N/A
0.06850.14380.6887
N/A
N/A
N/A
0.17550.11360.1755
N/A
N/A
0.59920.45390.43250.37970.3797
N/A
N/A
N/A
0.3040N/A
N/A
0.08470.1876
0.7266
N/A
N/A
N/A
0.18530.12200.1853
N/A
N/A
0.67680.56270.5367F
0.31410.3141P

0.25860.26080.23810.34440.08960.15770.56690.19090.13440.19090.49180.39030.37470.15590.15590.18380.02780.03090.19430.1896
0.16210.20280.20010.22790.24570.25020.06870.06830.06570.13330.06490.1066
0.26820.10570.08170.10570.26070.22820.21980.05240.00050.00100.05690.04840.03590.05700.0576
0.06370.0687
0.0668

Table 14: English Opinion Holder and Opinion Target results
Group
ICU
ICU
kle
kle
TUT
TUT

Type
L
S
L
S
L
S

P

0.40000.13330.39230.1250HolderR
NA
NA
0.5076
0.53220.28330.2829F

0.44740.21320.32900.1735P
0.10590.0374

TargetR
0.17610.1793
NA
NA
NA
NA

F
0.13240.0618PolarityR
N/A
N/A
0.43010.31590.5536
N/A
N/A
N/A
N/A
N/A
N/A
0.24130.05940.06990.18300.11420.15270.43940.4336
0.36710.31930.3100N/A
N/A
0.46450.33650.6209
N/A
N/A
N/A
N/A
N/A
N/A
0.27960.00470.00950.21800.11850.13740.50240.50710.41710.3602
0.3365

F

0.32300.28570.33300.20870.03790.04280.18850.14250.15730.27750.27380.28120.2777
0.27690.11970.11360.11880.08830.00100.00190.09030.0687
0.05690.10240.10340.11050.11460.1115

Table 15: Japanese opinionated/relevance/polarity analysis results
Group

RunID

EHBN
HCU
HCU
MIRAC
NAK
NAK
NAK
NLCL
TAK
TUT∗
TUT∗
UniNe
EHBN
HCU
HCU
MIRAC
NAK
NAK
NAK
NLCL
TAK
TUT∗
TUT∗
UniNe
∗

L/S

1
1
2
1
1
2
3
1
1
1
2
1
1
1
2
1
1
2
3
1
1
1
2
1
=

L
L
L
L
L
L
L
L
L
L
L
L
S
S
S
S
S
S
S
S
S
S
S
S

P
0.4921
0.619
0.77540.316
0.81150.7886
0.78130.42550.51910.6742
0.6742
0.53630.37380.48940.65440.24120.68850.66120.65740.31350.4166
0.5416
0.5416
0.4164

OpinionatedRF
0.73130.58830.51380.56150.21110.33190.08940.13940.34160.48080.30920.44420.36330.496
0.22340.293
0.27980.3636
0.562
0.613
0.562
0.613
0.19990.29120.76270.50170.5577
0.52130.2446
0.3561
0.09360.13490.39790.50430.35590.46270.41970.51230.226
0.26270.30830.35440.61990.57810.61990.57810.21310.2819P
0.48190.45450.53670.55270.55270.41470.28080.22330.301
0.30620.30620.1553

Relevance
R
0.6354
N/A
N/A
0.0816
N/A
N/A
N/A
0.1891
N/A
0.29250.29250.19180.7321
N/A
N/A
0.0821
N/A
N/A
N/A
0.2107N/A
0.3357
0.33570.1464

F
0.5481P

0.13840.24650.49220.49770.49340.27970.38250.38250.26230.40590.1201

0.46380.4596
0.42830.3251

0.2394
0.49330.50620.50220.24790.32030.32030.15070.51720.48060.45350.2914

Polarity
R
N/A
N/A
N/A
0.01830.1801
0.167
0.1936
N/A
0.11380.214
0.19940.0548N/A
N/A
N/A
0.01660.21540.19980.2232
N/A
0.13160.24170.22810.0497F

0.03410.26370.25010.27810.18280.292
0.2721
0.09380.031
0.29990.28650.309
0.20980.32160.30350.0849organizer

Table 16: Traditional Chinese opinionated/relevance/polarity analysis results
Group
WIA
CityUHK
CityUHK
CityUHK
iclpkuiclpku
NLCL
NLCL
NLCL
NTU∗
NTU∗
NTU∗
TTRD∗∗
TTRD
UniNe
WIA
CityUHK
CityUHK
CityUHK
iclpkuiclpku
NLCL
NLCL
NLCL
NTU∗
NTU∗
NTU∗
TTRD∗∗
TTRD
UniNe
∗
∗∗

=
=

RunID
1
1
2
3
1
2
1
2
3
1
2
3
1
2
1
1
1
2
3
1
2
1
2
3
1
2
3
1
2
1

L/S
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S

organizer
late submissionP
0.72980.66010.74320.65200.70150.58120.53580.47600.49440.56480.55750.55750.51100.56640.54280.85200.83640.90030.81780.85670.74230.62590.58300.60050.7076
0.69780.69780.64520.73840.6921

OpinionatedRF
0.52110.60800.84460.74110.6526
0.69500.86980.74530.62790.66260.73830.65040.2676
0.35700.74150.57980.50640.50030.89690.69310.88680.6846
0.88680.6846
0.93450.66070.66220.6106
0.92670.68460.60030.70430.90370.86870.73330.80820.92200.86680.69980.7704
0.7866
0.76380.29300.39910.74120.6526
0.52550.56050.93070.80400.9172
0.79260.9172
0.7926
0.93950.76500.67440.70500.93790.7965

P
0.99490.99430.99430.92400.92830.92980.96150.98040.98070.96730.96600.96140.97880.95300.95300.84870.85730.86400.88490.91990.91230.89920.88850.8746

Relevance
R
0.5306
N/A
N/A
N/A
0.67680.67680.1801
0.48460.34070.71030.64480.63240.84130.89670.8456
0.4061
N/A
N/A
N/A
0.56260.56260.14540.41100.28630.64370.58900.58490.80440.86760.8443

F
0.69210.80540.80540.30150.63680.49870.81700.77800.76890.89990.93000.89980.57400.70750.70750.24820.55560.43010.74530.71820.71280.84910.87790.8592

Polarity
S-P
0.69310.53610.51970.5053
0.48100.45130.48750.4796
0.48110.37470.46510.42930.70030.54630.52880.52590.50850.49090.49790.48090.48440.39240.47580.4431

Recall-based Polarity
PR
F
0.50580.36110.42140.35390.45280.39730.38620.3391
0.36120.32940.43950.3766
0.33740.30200.31870.26230.33320.2935
N/A
N/A
N/A
0.27530.43720.33790.26740.42530.32830.26820.42670.32940.19150.35010.2476
0.26340.30800.28400.23300.39780.29390.59660.4204
0.49320.45690.49360.4746
0.47610.38770.42740.4301
0.48490.45580.43570.35590.39180.36440.38610.3750N/A
N/A
N/A
0.35230.46340.40030.3356
0.4411
0.3811
0.33800.44430.38390.25310.36860.30020.35140.32090.33540.30670.41560.3529

Table 17: Traditional Chinese opinion holder/target analysis results
Group
WIA
iclpkuiclpku
NTU∗
NTU∗
NTU∗
TTRD∗∗
TTRD
WIA
iclpkuiclpku
NTU∗
NTU∗
NTU∗
TTRD∗∗
TTRD
∗
∗∗

=
=

Run
ID
1
1
2
1
2
3
1
2
1
1
2
1
2
3
1
2

L
/S
L
L
L
L
L
L
L
L
S
S
S
S
S
S
S
S

P
0.8254
0.58720.59880.50280.45870.31910.56450.59470.82380.57970.5816
0.48250.43580.29690.5496
0.5840Holder-TR
0.8254
0.58720.59880.50280.45870.31910.56450.59470.82380.57970.5816
0.48250.43580.29690.5496
0.5840F
0.8254
0.58720.59880.50280.45870.31910.56450.59470.82380.57970.5816
0.48250.43580.29690.5496
0.5840P
0.29920.20510.17330.14140.12730.08860.14430.16780.19880.13030.10350.08140.07230.04930.0822
0.0972P
0.60580.02130.01990.03580.1066
0.6331
0.02280.02230.03720.1356

Target-TR
0.60580.02130.0199N/A
N/A
N/A
0.03580.1066
0.6331
0.02280.0223
N/A
N/A
N/A
0.03720.1356F
0.60580.02130.0199P
0.21960.00740.00580.03580.1066
0.6331
0.02280.02230.00910.03010.15280.00510.00400.03720.13560.00560.0226

Target-RB
R
0.31600.01330.0147
N/A
N/A
N/A
0.03400.07180.38060.01590.0175
N/A
N/A
N/A
0.03580.0932organizer
late submission

The ICU group participated in the opinionated,
relevance, and target identiﬁcation tasks. They
use the KLDivergence statistic to identify words
that are discriminative for opinionated or nonopinionated sentences, and use tri-gram language
models for each topic with web snippets used forquery expansion for relevance. They use a syntactic parse and distance from potential opinion
targets in the parse tree with a machine learning
framework over the parse constituents to identify
targets.
The SICS team use only general linguistic information and dependency parsing to extract three
diﬀerent types of features from the NTCIR-6 data
for an SVM-based approach. They present an interesting investigation of feature selection for theopinionated sentence detection task.

7.2

Holder-RB
R
F
0.43050.35310.36840.26350.44200.24900.45080.21530.4066
0.19390.28290.13490.53730.22750.4002
0.23650.49520.28380.40530.19720.45700.16880.44900.13780.39970.12250.27230.08340.52950.14230.40130.1565

Japanese

Eight teams participated in the task at Japaneseside. We introduced their systems by dividingthem into two parts: (1) four multilingual participants by focusing on language portable approachand (2) four Japanese native participants by focusing on opinion clues from Japanese native viewpoints.
Four teams of Japanese side participants
challenged at multilingual sides with language
portable approach.
The Sussex University(NLCL) team did not use preliminary word segmentation in Japanese and Chinese. They used
the same routine for ﬁnding basic lexical units
in all languages. They extracted opinion bearing
terms using χ2 score. Their results improved withthe manual list plus all associated words. University of Neuchâtel (UniNe) team utilized bigram
indexing scheme for Japanese and suggested using a statistical method (Z score) to identify theterms that adequately characterize subsets of the
corpus belonging to positive, negative, neutral or
non opinionated subsets. MIRACLE team is a
research consortium formed by research groups ofthree diﬀerent universities in Madrid. They implemented a semantic knowledge base using machinetranslated dictionary. They provided some interesting approaches for non-native speakers to
implement a multilingual opinion extraction system. A Japanese side organizer also submitted
the results as Toyohashi University of Technology(TUT) team in Japanese and English. The feature
selection was based on statistical χ-square test andimplemented polarity classiﬁcation systems using
multi-label classiﬁcation methods.
Other four Japanese participants focused onJapanese oriented clues to extract opinion or classify polarity. NEC (EHBN) focused on consecutive property of opinionated or relevant sentencesin Japanese and proposed Sliding Window Framework and proved to obtain high recall. Hiroshima
City University (HCU) implemented their opinion extraction system with 760,000 sentence-ﬁnal
expressions collected from their newspaper corpus and provided the detailed analysis for theirresults. Keio University (NAK) also focused on
auxiliary verbs and utilized some other features
such as character types to extract opinions andclassify the polarity. Sokendai University (TAK)
implemented a polarity classiﬁcation system usinga small number of signpost expressions.

7.3

Traditional Chinese

Seven teams participated in the task at traditional
Chinese side. In this section, we introduced only
participants who participated in both the traditional and simpliﬁed Chinese tasks, or only in the
traditional Chinese task, in alphabetic order. De-F
0.25910.00950.00830.01440.04240.21810.00770.00650.0096
0.0363

scriptions of other multilingual participants’ systems (NLCL and UniNe) are already provided by
the Japanese side.
CityUHK: Language Information Sciences Center, City University of Hong Kong In their system,
supervised approaches and ensemble techniques
have been used and compared in our participating system. Two kinds of supervised approaches
were employed: 1) the supervised lexicon-based
approach, 2) machine learning approaches, andensemble techniques were also used to combine
the results given by diﬀerent approaches. Threeclassiﬁers, the supervised lexicon-based classiﬁer,
SVM classiﬁer, and Bayes classiﬁer, are adopted
in their system.
iclpku: Institute of Computational Linguistic,
Peking University In their system, maximum entropy model is used to predict the polarity class.
A rule-based pattern matching scheme is devised
to ﬁnd topic-relevant sentence. For the subtask ofdetecting holders and targets, the CRF model isadopted.
NTUCopeOpi: National Taiwan University
They adopted their opinion analysis system
CopeOpi to analyze opinionated information in
NTCIR-7 MOAT tasks document collections. For
opinion extraction task, their algorithm was basedon the bag-of-character methods proposed in
NTCIR-6 and considered morphological structures of Chinese words to extract opinion words
correctly. How distant an opinion word is to theend of the sentence was also considered to adjust
its opinion weight. For the relevance judgmenttask, the distance of two sentences were also considered as a factor in their weighting formula.
TTRD: Tornado Technologies Co.
Theirmethod for opinion analysis tasks involves two different approaches: (1) the machine learning-based
prototype system (on the basis of support vector
machines (SVMs)) and (2) stochastic estimationof the character-level of words. For relevance judgment, they adopt lemur as their language modeland Tornado Search 5.0 to perform the second retrieval. Two-stage Dirichlet smoothing strategy isthen applied.

pated both the traditional and simpliﬁed Chinesetasks ( iclpku, NTUCopeOpi, TTRD) are alreadyprovided by the traditional Chinese side.
ISCAS: Institute of Software, Chinese Academyof Sciences For identifying the opinionated sentences, in their system an EM algorithm is used
to extract the sentiment words based on the sentimental dictionary, and then an iterative algorithm is used to estimate the score of the sentimentwords and the sentences. In relevant sentences detection sub-task, pseudo feedback and query extension methods are used based on the traditional
IR model.
NEU: Northeastern University Their system
adopts a sentiment lexicon-based (SLB) approach
to identifying opinionated sentences. Their Chinese sentiment dictionary is extracted from thefamous Chinese concept lexicon Hownet.
NLPR: Institute of Automation, Chinese
Academy of Sciences In their system the domain
adaptation technique is used for identifying the
subjective sentences and the data in NTCIR6 isused for training subjective classiﬁer. In order
to extract the opinion holder, they use the CRF
model, which is combined with manual designed
heuristics rules. The features in their CRF model
include part-of-speech features, semantic class features, contextual features, and dependency features through parsing analysis.
WIA: Chinese University of Hong Kong They
adopt a multi-pass coarse-ﬁne analysis strategy for
detecting opinionated sentences. A base classiﬁer
ﬁrstly coarsely estimates the opinion of sentences
and the document. The obtained document-leveland sentence-level opinions are then incorporated
in a complex classiﬁer to analyze the opinion ofsentences again to generate reﬁned sentence anddocument opinions. The updated opinion features
are feed back to the classiﬁer to further reﬁnethe opinion analysis. Such circles terminate until the analysis results converge. Similar strategy
is adopted to sentence-topic relevance estimation.
Furthermore, the mutual reinforcement between
the analysis of sentence relevance and sentence
opinion are integrated in one framework.

7.4

8Conclusions

8.1

Overview of results in NTCIR-7

Simplified Chinese

Nine groups submitted runs for the simpliﬁed Chinese evaluation. It’s a pity that one group giveup to submit the technical paper. In this section, we introduced the system who participatedin the simpliﬁed Chinese task and the system are
not described by others in alphabetic order. Descriptions of the multilingual participant’s system(NLCL) are already provided by the Japaneseside. Description of other systems who partici-

In NTCIR-7 MOAT, we have several new challenging points diﬀerent from the ﬁrst NTCIR-6 OAT
as follows:
1. The participants could use NTCIR-6 OAT
corpus: large size test collection with detailed
annotation appropriate for training use.

2. The task focused on not only sentence-levelannotation but also opinion expression (subsentence/clause) level annotation.
3. The number of teams who participated at
multilingual sides with language portableapproaches increased (two teams ⇒ eightteams).
4. A new target language (Simpliﬁed Chinese)and a new subtask (opinion target detection)
were added to the scope of MOAT.
Basically, the evaluation results of the participants improved a lot based on the ﬁrst point
and the mature of the opinion analysis technology. We also succeeded in opinion expression (subsentence/clause) level annotation using annotation tools and attained high κ coeﬃcient for theassessors’ agreements. For the third and fourth
points, many pariticpants challenged their new exciting approach this year and provided new precious insights.

8.2

Directions for next challenge

Instead of sudden change of the target document
genre, 21 teams participated in the MOAT thisyear. We greatly appreciate their eﬀorts and try
to advance the community for the next challenge.
We took the legally safe approach this year withconsidering the present state of the research community, but we also understood the value of opinion analysis of the user generated contents suchas blogs deeply. We struggle to solve the problem
and hope to collaborate with participants to breakout a new challenge.

Acknowledgements
We greatly appreciate the eﬀorts of all the participants in the Multilingual Opinion Analysis Task
at the Seventh NTCIR Workshop.

References[1] Y. Choi, C. Cardie, E. Riloﬀ, and S. Patwardhan.
Identifying sources of opinions with conditionalrandom ﬁelds and extraction patterns. In Proc. ofthe 2005 Human Language Technology Conf. andConf. on Empirical Methods in Natural Language
Processing (HLT/EMNLP 2005), Vancouver, B.
C., 2005.
[2] M. Gamon and A. Aue. Proc. of Wksp. on Sentiment and Subjectivity in Text at the 21th Int’l
Conf. on Computational Linguistics / the 44th
Ann. Meeting of the Assoc. for Computational

Linguistics (COLING/ACL 2006). The Association for Computational Linguistcs, Sydney, Austraria, July 2006.
[3] L. W. Ku, T. H. Wu, L. Y. Lee, and H. H. Chen.
Construction of an Evaluation Corpus for Opinion Extraction. In Proc. of the Fifth NTCIR
Wksp. on Evaluation of Information Access Technologies: Information Retrieval, Question Answering, and Cross-Lingual Information Access,pages 513–520, December 2005.
[4] National Institute of Informatics. NTCIR (NII
Test Collection for IR Systems) Project [online].
In NTCIR (NII Test Collection for IR Systems)
Project website, 1998-2008. [cited 2008-10-10].
Available from: <http://research.nii.ac.jp/ntcir/>.
[5] National Institute of Standars and Technology.
TREC (Text REtrieval Conference) 2006-2008:
BLOG Track [online]. In TREC-BLOG Information Retrieval Wiki, 2008. [cited 2008-10-10].
Available from: <http://ir.dcs.gla.ac.uk/wiki/TRECBLOG>.
[6] B. Pang and L. Lee. Opinion Mining and Sentiment Analysis. Now Publishers Inc, July 2008.
[7] J. Ruppenhofer, S. Somasundaran, and J. Wiebe.
Finding the Sources and Targets of Subjective
Expressions. In Proceedings of the Sixth International Language Resources and Evaluation(LREC’08), Marrakech, Morocco, May 2008.
[8] Y. Seki, K. Eguchi, and N. Kando. Multidocument viewpoint summarization focused on
facts, opinion and knowledge. In J. G. Shanahan, Y. Qu, and J. Wiebe, editors, Computing
Attitude and Aﬀect in Text: Theory and Applications, volume 20 of The Information Retrieval
Series, chapter 24, pages 317–336. SpringerVerlag, New York, December 2005.
[9] Y. Seki, D. K. Evans, L. W. Ku, H. H. Chen,
N. Kando, and C. Y. Lin. Overview of Opinion
Analysis Pilot Task at NTCIR-6. In Proc. of the
Sixth NTCIR Workshop Meeting on Evaluationof Information Access Technologies: Information Retrieval, Question Answering, and CrossLingual Information Access, pages 265–278, NII,
Japan, May 2007.
[10] J. G. Shanahan, Y. Qu, and J. Wiebe. Computing Attitude and Aﬀect in Text: Theory andApplications, volume 20 of The Information Retrieval Series. Springer-Verlag, New York, December 2005.
[11] The Association for the Advancement of Artiﬁcial Intelligence. International Conference on
Weblogs and Social Media. In Proc. of thethird Int’l AAAI Conference on Weblogs and Social Media (forthcoming), San Jose, California,
March 2009. [cited 2008-10-10]. Available from:<http://www.icwsm.org/2009/index.shtml>.
[12] J. Wiebe, T. Wilson, and C. Cardie. Annotating
Expressions of Opinions and Emotions in Language. Language Resources and Evaluation, 39(23):165–210, 2005.
[13] J. M. Wiebe, T. Wilson, R. F. Bruce, M. Bell,and M. Martin. Learning subjective language.
Computational Linguistics, 30(3):277–308, 2004.

[14] T. Wilson, J. Wiebe, and P. Hoﬀmann. Recognizing contextual polarity in phrase-level sentiment analysis. In Proc. of the 2005 Human
Language Technology Conf. and Conf. on Empirical Methods in Natural Language Processing(HLT/EMNLP 2005), Vancouver, B. C., 2005.