Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Supervised Approaches and Dependency Parsing for
Chinese Opinion Analysis at NTCIR-8
Bin LU

Benjamin K. TSOU

Tao JIANG

Department of Chinese, Translationand Linguistics, & Language
Information Sciences ResearchCentre, City University of Hong
Kong

Language Information Sciences
Research Centre, & Department ofChinese, Translation and
Linguistics, City University of Hong
Kong

ChiLin Star Corp, Southern
Software Park, Zhuhai, China &lubin2010@gmail.com

rlbtsou@gmail.com

ABSTRACTIn this paper, we describe our participating system, which is basedon supervised approaches and dependency parsing, for opinion
analysis on traditional Chinese texts at NTCIR-8. For opinionatedsentence recognition, the supervised lexicon-based approach,
SVM and Maximum Entropy are combined together. For polarity
classification, we use only the supervised lexicon-based approach.
For opinion holder and target identification, we, on the basis ofdependency parsing, identify opinion holders by means ofreporting verbs and identify opinion targets by considering both
opinion holders and opinion-bearing words. The results show thatamong all the teams participating in the traditional Chinese task,
our system achieve: 1) the highest F-measure on the opinionatedsentence recognition task, 2) the second highest F-measure on theidentification of both opinion holders and targets, 3) the middle
ranking for opinion polarity classification.

Categories and Subject Descriptors
I.2.7 [Artificial Intelligence]: Natural Language Processing – text
analysis

General Terms
Algorithms, Experimentation, Languages

Keywords
Opinion Analysis, Machine Learning, Supervised Approaches,
Dependency Parsing

1. INTRODUCTION
In recent years, opinion analysis, which mines opinions frominformation sources such as news, blogs, and product reviews, has
drawn much attention in the NLP field [3], [17], [27], [4], [16]. It
has many applications such as social media monitoring, market
research, and public relations.
To analyze the opinions in news text, we first need to distinguishbetween opinions and facts, which could be done in the sentencelevel, i.e. to identify opinionated sentences from news articles,
which is also called subjectivity classification. When an opinion isfound in a sentence, it may involve different components, namely

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,requires prior specific permission and/or a fee.
The NTCIR-8 Workshop, June 15–18, 2010, Tokyo, Japan.
Copyright 2010.

Northeastern University, Shenyang,
China

jiangtaoster@gmail.com

opinion expression, opinion holder and opinion target [28]. An
opinion expression could be related to polarity (positive, negative
or neutral) and intensity; an opinion holder is usually an entitythat holds an opinion, and an opinion target is what the opinion isabout [5], [6]. For more related work on opinion holder/target
identification, please refer to [1], [2], [7], [8], [13], [19],[22], [25],[26].
The NTCIR-8 Multilingual Opinion Analysis Task (MOAT) [23]provides an opportunity to evaluate the techniques used bydifferent participants based on a common evaluation frameworkin Chinese (simplified and traditional), Japanese and English,
following the opinion analysis pilot task (OAPT) at NTCIR-6 in2007 [20] and MOAT at NTCIR-7 in 2008 [21], [30], [8].
Participants need to tag opinions at a sub-sentence level andannotate the opinion features, including subjectivity, relevance,
polarity, opinion holder and opinion target. More descriptions
about the task are covered in the overview paper [23].
This paper describes the CityU (HK)’s system used in the
traditional Chinese task at NTCIR-8 MOAT. We participated infour of the five subtasks: opinionated sentence recognition(subjectivity recognition), opinion polarity classification, opinion
holder identification and opinion target identification. Three runs
were submitted for the subtasks. For opinionated sentence
recognition, the supervised lexicon-based approach, SVM andMaximum Entropy were combined together. For polarity
classification, we just use the supervised lexicon-based approach.
For opinion holder and target identification, based on dependency
parsing, we identify opinion holders by means of reporting verbs
and identify opinion targets by considering both opinion holders
and opinion-bearing words. The results show that among all theteams participating in the traditional Chinese task, our systemachieved 1) the highest F-measure on the opinionated sentence
recognition task, 2) the second highest F-measure on theidentification of both opinion holders and targets, 3) the middle
ranking for opinion polarity classification.
The rest of this paper is organized as follows. Section 2 presents
linguistic analysis of opinions. The supervised approaches torecognize opinionated sentences and to determine polarity aredescribed in Section 3, followed by the dependency parsing-based
approach for identifying opinion holders and targets in Section 4.
Section 5 gives the evaluation results and finally, Section 6
concludes this paper.

2. LINGUISTIC ANALYSIS OF OPINIONS― 234 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

USA, UK, France, China and Russia in a
23-points
declaration, advocate to enhance the 1972’s anti-trajectory
missile treaty ,2.1 Subjectivity and Polarity
The opinions in news text maybe explicitly mentioned, or they
may be expressed indirectly by the types of words and the style oflanguage that a speaker or writer uses [28]. Three kinds of lexicalclues were exploited for opinionated sentence recognition:
1)
2)

3)

Reporting verbs: verbs indicating speech events, such asᎅ(say), ᣛߎ(point out), 䁡⠆ (think), etc.
Sentiment-bearing items: words/phrases bearing polarity(i.e. positive, negative or neutral), such as ‫( ړ‬good), ।ཆ (praise), ᙑᎄ (wrong), etc.
Adverb clues: adverbs frequently co-occurring with
opinions, such as ‫ױ‬౨ (perhaps), ॺൄ (very much), ᄕ৫(extremely), etc.

In addition to the three kinds of clues, two other kinds were used
for polarity classification:
4)

Negation marker: words used to reverse the polarity of apolar item, such as լ (no), լམ(had not), լᄎ (will not),
etc.

5)

Discourse marker: discourse markers that may reverse
the polarity of previous clause(s), such as ឈྥ (although),լመ (but), ‫܀‬ਢ(but), etc.

2.2 Opinion Holders
Opinion holders are usually named entities, including personnames, organization names, and personal titles. Some examples of
opinion holders are as follows:
Person Names: ᆖᛎᖂ୮ᑛዿ (economist Ol);
Organization Names: ૎ഏਙࢌ (UK government);
Personal Titles: ᆖᛎᖂ୮(economists);
Opinion holders can also be common noun phrases, such as ᐗ೸(companies), and ࠟՏ‫ټ‬ᖂ‫( س‬two thousand students).
Pronouns can also be opinion holders, e.g. ‫( ה‬he), ‫ה‬ଚ (they),‫(ݺ‬I), ‫( ܃‬you), etc. Sometimes, though not very often, nounphrases with relative clauses can also be opinion holders, such as‫്׌‬ᤕশ‫אڮ‬ฤ‫ٽ‬ᖆऄֱ‫ڤ‬ՀፕऱԳ (People advocating that
Suharto should step down under a way complying with theConstitution).

The opinion target in sentence a) is ‫ף‬㺞ԫ԰ԮԲ‫ڣ‬ऱ֘ᐘሐଆᐘ ය પ  (to enhance the 1972’s anti-trajectory missile treaty),
which is a verb phrase.
Occasionally, different readers may have different views on what
the holder and target are in a sentence. Consider the following
sentence:b) ‫܀‬ભഏ ᜔อؒ‫ݦ‬լᣤૹ চஇ ຍႈ යપ ऱ ய࣠ Δ

There can be some degree of structural ambiguity, and thesentence can be rendered into English as b.1) or b.2), because
Chinese could be more free in terms of syntax.
b.1) But that US President Bush does not accept this treaty
severely impairs the effects of this treaty.
b.2) But US President Bush does not accept this treaty, and he(it) severely impairs the effects of this treaty.
In sentence b.1), the opinion target could be the italicized
embedded clause, while in sentence b.2) the target could be he in
the second part referring to ભഏ᜔อؒ‫( ݦ‬US President Bush) in
the first part, or could be ຍႈයપऱய࣠ (the effects of thistreaty, or even could be ຍႈයપ (this treaty).
Meanwhile, the opinion target is sometimes dependent on the
choice of opinion holder. E.g. in sentence b.1), if we consider ભഏ᜔อؒ‫(ݦ‬US President Bush) is the opinion holder, the target
could be ຍႈයપ (this treaty), which is different from the clausetarget mentioned above.

2.4 Dependency Parsing and Opinion Holders /Targets
Dependency representation provides a simple description of thegrammatical relationships in a sentence [11]. It represents all
sentence relationships uniformly as typed dependency relations
between pairs of words. Some major dependency relations for
Chinese are shown in Table 1 (Please refer to [12], [14] for moredetailed description of dependency relations in Chinese).
Table 1. Some Dependency Relations in Chinese
RelationЏ䇧 (Subject-Verb )ࡼᆒ (Verb-Object)ᅮЁ (Attributive-Noun)᭄䞣 (Quantifier)⣀ゟ㒧ᵘ(Independent structure)ࠡ䰘ࡴ (Left Adjunct)

2.3 Opinion Targets
While opinion holders usually are agents or entities who
may bear opinions, opinion targets could be more abstract,
including but not limited to agents, concrete objects,
actions, events or even abstract ideas. Thus, opinion targets
are much more diverse with respect to its forms. In addition
to these forms of opinion holders, opinion targets could
also be verb phrases or embedded clauses. Consider thefollowing Chinese sentence:൷࠹ ຍႈ යપ Δ

Symbol
SBV
VOB
ATT
QUN
IS
LAD

Consider the following Chinese sentence:a) ભഏΕ૎ഏΕऄഏΕխ٥ ࡉ ঎ᢅཎ ‫ ڇ‬ԫٝ ֥Կរ ऱᜢࣔ խ Δ‫ף ്׌‬㺞!ԫ԰ԮԲ‫ڣ‬ऱ֘ᐘሐଆᐘ
යપΔ― 235 ―

c) ঎ഏ
؆९ʳ
ْՅᘭ֛ʳ
ᎅʳ Δ‫ק‬પʳ ࣟʳ ‫ٻ‬ʳ
ឩ്!ʳ ਢʳ “ʳ ᝬ‫ٻ‬ʳ
ᙑᎄʳ ऱʳ ֱ‫ٻ‬ʳ ”ʳ Ζ
Russian Foreign Minister Ivanov said that NATO's eastward
expansion was "Towards the wrong direction."

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Figure 1. Dependency Tree for Sentence c)
Its dependency tree is shown in Figure 1. The head of the wholesentence is the verb ᎅ (said), whose subject and object are
respectively ঎ 㧺 ؆ 叿 ْ Յ 册 ֛ (Russian Foreign Minister
Ivanov) and the embedded clause ‫ק‬પࣟ‫ٻ‬ឩ്ਢ“ᝬ‫ٻ‬ᙑᎄऱֱ ‫(” ٻ‬NATO's eastward expansion was "Towards the wrong
direction."). The reporting verb ᎅ (said) indicates a speech event
expressing an opinion given by the holder ঎㧺 ؆叿 ْՅ册ᄦ(Russian Foreign Minister Ivanov). Meanwhile, the opinionbearing word ᙑᎄ (wrong) shows a negative attitude towards theopinion target ‫ק‬પࣟ‫ٻ‬ឩ്!)NATO's eastward expansion*.

3. SUBJECTIVITY
CLASSIFICATIONANDPOLARITY

Therefore, we assume that a large proportion of opinion holders
are governed by such reporting verbs, while opinion targets areusually governed by opinion-bearing words/phrases.

The motivation of our system is to make full use of both manual
labeled lexicons and annotated corpora to improve opinion
analysis. The combined lexicon is introduced above, and thetraining data includes the traditional Chinese data from NTCIR-6
OAPT and NTCIR-7 MOAT, as well as the traditional Chinese
sample data from NTCIR-8 MOAT. The Chinese sentences aresegmented into words using a production segmentation system.
Words are treated as basic features for opinion analysis in
traditional Chinese texts. The supervised lexicon-based approachand machine learning approaches, as well as the ensemble method
to combine them, are described in the next subsections,
respectively.

2.5 Lexicon Preparation

3.1 The Supervised Lexicon-based Method

Over the past years, several traditional Chinese resources of polar
items have been collected for opinion analysis in our center,
including NTU Sentiment Dictionary (NTUSD)1, The Lexicon ofChinese Positive Words (LCPW) [24], The Lexicon of Chinese
Negative Words (LCNW) [31], and CityU’s polar word/phrase
list, which were manually marked in the political news data by
trained annotators from our center. Polar items manually marked
with the SENTIMENT_KW tag (SKPI), including only positive
and negative items but not neutral ones, are also extracted fromthe Chinese sample data of NTCIR-6 OAPT [20]. All these polar
lexicons are combined to get a large lexicon2 consisting of 13,437
positive items and 18,365 negative items, a total of 31,802 items.

For opinionated sentence recognition, the lexicons of reporting
verbs and polar items mentioned above are used after adjusted on
the training data. For the adjustment of lexicons, we compute thepercentage of each reporting verb / polar item occurring inopinionated sentences over all sentences in the training corpus. If
a reporting verb has a lower percentage than a threshold Ԧreporting,it would be removed from the lexicon of reporting verbs.
Similarly, a polar item would be removed from the lexicon ofpolar items, if it has a lower percentage than the threshold Ԧpolar.
We optimize Ԧpolar and Ԧreporting by using a brute-force grid searchwithin the range [0-100] (the step size is 1). More details of the
training algorithm is described in [8].

The reporting verbs were firstly collected from the Chinese
sample data of NTCIR-6 OAPT in which the OPINION_OPR tag
was used to mark them, and then was extended to 308 from 68
words through manual synonym search in HowNet3, WordNet andTongyici Cilin [15]. Some frequently used reporting verbs
include ᎅ(say), 㸼⼎(express), ᣛߎ(point out), 䁡⠆ (think),ᔎ䂓(stress), ৠᛣ(agree), ⱐ㸼(publish), etc. Some of thereporting verbs could also convey opinions, such as ᡍ䀩(criticize), 䅈䊀 (condemn), ᣛ䊀 (denounce), 䅮᦮ (praise), etc.
There were 48 negation markers manually collected by inspectionof HowNet and Tongyici Cilin, and a small set of discourse
markers (i.e. 5 single markers and 7 pairs of markers) which may
reverse the polarity of previous contexts were also collected [8].

There are two reasons for this adjustment: a) lexical items werecollected from multiple sources which have not been cleaned, andcould contain errors or typos, especially the polar items in SKPI,
such as 䱣ⴔ (with), ৃҹ (be able to), etc.; b) these items marked
by annotators with their own subjectivities could be contextual or
not suitable for news domain. Two kinds of items could be
filtered out during the tuning process: 1) the noisy terms whichwere actually not reporting verbs or polar items, e.g. 㾔‫ܝ‬(sightseeing) in LCPW, ᅮϟ (set) and ࠡᕔ (head for) in SKPI;
2) the reporting verbs or polar items which may present facts andfrequently occur in factual sentences, e.g. ᲈ䲼 (downpour) in
NTUSD and 㽆᪞ (attack) in NTUSD and CPWP.

1

http://nlg18.csie.ntu.edu.tw:8080/opinion/index.html

2

An item could be positive and negative at the same time, and wetreat it as two items. For example, ߻ַ is marked as positive in
NTUSD, and negative in SIST.

3

http://www.keenage.com

Given a segmented sentence, we check whether a polar item(including adverbs) or a reporting verb occur in it. If a sentence
contains at least one item in the remaining polar item lexicon(including the adverb clues) or the reporting verb lexicon, it isreported as opinionated, otherwise not opinionated.
The adjustment process of lexicons for opinion polarity
classification is similar to that for opinionated sentence
recognition except that the reporting verbs are not used for― 236 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

opinion polarity classification. The polar item lexicon is used
together with negation markers and discourse markers. The
polarity of a sentence is calculated by checking the remaining
polar items. A polar item gets a score of 1 if it is marked as
positive in the lexicon, -1 if negative, and 0 if both positive andnegative. The scores are summed up, and the sentence is reported
as positive if the sum is bigger than 0, negative if smaller than 0
and neutral equal to 0.

3.2 Machine Learning Approaches
The opinionated sentence recognition task is a sentenceclassification problem with only two labels, opinionated or not
while the opinion polarity classification is a three-class sentenceclassification problem with three labels, positive, negative or
neutral. The unigrams of Chinese words are used as the linguisticfeature for machine learning.
Maximum Entropy (MaxEnt) and support vector machines (SVM)
are explored. The models are trained on the training data, and thenapplied to classify new test data. Note these models are only used
for the opinionated sentence recognition task, which is different
from [8].
We constructed one feature vector for each sentence withunigrams as features and the frequency of terms as the weight of
the feature, and the feature vector was normalized and then fed
into the classifiers for learning. Joachim's SVMlight package4 andLe Zhang’s maximum entropy tool 5 are used for training andtesting.

3.3 Combination
After obtaining the set of classifiers (e.g. the supervised lexiconbased classifier, SVM classifier and MaxEnt classifier), we could
exploit different ensemble methods to combine the results ofindividual classifiers. The commonly used ensemble strategies
include majority voting, sum, product, max, min, etc [18], [28].
The effectiveness of combination is determined by the diversity of
its component classifiers [18].

4.1 Preprocessing of Chinese Sentences
Since there is no space between words in Chinese, if namedentities including person names, organization names, and place
names within sentences cannot be identified and are split into
single characters, it would be quite impossible for the parser tocorrectly parse the sentence. Thus, named entities are firstrecognized with a traditional Chinese word segmentation toolwith access to the LIVAC7’s dictionary containing more than 1.5
million word entries collected from traditional Chinese news
published in Hong Kong and Taiwan, and it should have better
coverage for words and named entities in traditional Chinese news
documents than the HITIR’s NLP package which is developed
within the simplified Chinese context. The identified namedentities, as well as the collected reporting verbs and opinionbearing words are added into the user dictionary of the HITIR’s
NLP package to help the Chinese parsing.
Another factor we investigated is parentheses: before parsing, theparentheses enclosing only English words or numbers are
removed in sentences, because the parser cannot properly process
the parentheses which may greatly influence the parsing result.

4.2 Identifying
Opinion
Reporting Verbs

Holderswith

4.2.1 Holder Candidate Generation
By intuition, we hypothesize that: 1) the subject (SBV) of
reporting verbs will be the opinion holders in opinionatedsentences; 2) if no reporting verb occurs in the opinionatedsentences, the author could be the opinion holder. In addition to
the hypothesis above, the following heuristic rules are used for
opinion holder identification:
1) Other words having relations with reporting verbs
If the subject of reporting verbs is not found in the sentence, butreporting verbs occur in the opinionated sentences, we will findthe word having relationship of ATT (attributive), VOB (object)or IS (independent structure) with the reporting verbs identified.
The reason for this is that sometimes the parser may wrongly
marked the subject (SBV) relation as other relations. Thus we
use this to improve the robustness of the parser.

We just used the intuitive majority voting strategy for theopinionated sentence recognition task: if two of the threecomponent classifiers mark a sentence as opinionated, thesentence would be marked as opinionated, just like the lenient
standard for evaluation.

2) Colon processing in Headlines

4. IDENTIFYING OPINION HOLDERS
AND TARGETS WITH DEPENDENCYTREEIn news headlines, the author of news report usually omits thereporting verb because of the length limitation of headlines, anduses a colon to denote the relation between opinion holders andopinions, and in such cases we just pick up the noun before thecolons as the target candidate if no reporting verbs are found in
the headlines. E.g. in the headlines ᐰ ௅ Κ ᆖ ᛎ ‫ ګ‬९ ዽ ‫־‬(Morgan: Economic growth has been shut down) and ‫ଁ؍ٱ‬ᥞ‫ڮ‬Κ‫࣋ڻڍ‬㥯ਙ᧢ኆᦞᖲᄎ (Indonesia Vellanto: Give up theopportunity of seizing power by coups for many times), thenouns before colons ᐰ ௅ (Morgan) and ‫ ଁ ؍ ٱ‬ᥞ ‫ڮ‬(Indonesia Vellanto) are the opinion holders respectively.

This section introduces our approach of using dependency parsing
to identify opinion holders/targets. For the dependency parsing,
we use HITIR’s NLP package 6 to parse the simplified Chinese
sentences, which are converted from the traditional Chinese ones.
The NLP package is developed within the simplified Chinesecontext, and can provide word segmentation, POS-tagging,dependency parsing and for Chinese sentences.

3) Holder in the previous sentence4

http://svmlight.joachims.org/

5

http://homepages.inf.ed.ac.uk/lzhang10/maxent_toolkit.html

6

http://ir.hit.edu.cn/

If no opinion holder is found in the current clause and one
holder candidate is found in the previous clause, we just choose7

http://www.livac.org― 237 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japanthe opinion holder of the previous clause as the holder
candidate, because an opinion holder may express several ideas
by several consecutive sentences or clauses.

can find ‫ק‬પࣟ‫ٻ‬ឩ് (NATO's eastward expansion) as
the subject of the embedded clause by the verb ᰃ (was)which is in turn the object of the reporting verb ᎅ (said).

4.2.2 Holder Candidate Expansion
Through the procedure of candidate generation, we may find aholder candidate containing only one single word. But the holder
may be a word sequence instead of a single word. Thus we further
expand the holder candidates from the core head word by thefollowing rules:
1) Attributive modifier (ATT)
E.g. in sentence a) mentioned in section 2.2, the subject of thereporting verb ᎅ (said) is ْՅᘭ֛(Ivanov), which has theattributive noun ؆९ (Foreign Minister) modified further by
an attributive noun ঎ഏ(Russia). Therefore, the final extended
opinion holder would be ؆ ९ ْ Յ ᘭ ֛ (Russian Foreign
Minister Ivanov).
2) Quantifier modifier (QUN)
E.g. the quantifier modifier ຝ։(some) in the noun phrase ຝ։ ࠅ ੊ ഏ ୮ (some Asian countries) should be part of theopinion holder.
3) ࡉ/֗ (and/or)
E.g. The dependency tree for the sentence ᤕশ‫׼ࡉڮ‬؆ࠟ‫ټ‬૨ ֱ ല Ꮖ ‫ ٻ‬ᤕ ‫ ׬‬ᘭ ༼ ‫ נ‬...(Suharto and two other army
generals addressed to Sukarno …) is illustrated in Figure 2.
The subject of the reporting verb ༼‫( נ‬addressed) is ല Ꮖ(generals) which will be expanded to ‫׼‬؆ࠟ‫ټ‬૨ֱലᏆ(twoother army generals) after expanding by ATTs and QUNs.
However, here we still need to extend the holder across ੠(and),and the final holder is ᤕশ‫׼ࡉڮ‬؆ࠟ‫ټ‬૨ֱലᏆ (Suharto
and two other army generals).

b) If no target candidate is found in step a), we try to find a
subject (SBV) whose parent is an opinion-bearing word in
the latter part of the sentence after the reporting verb as thetarget candidate.
2) If no target candidate is found in step 1), and no opinion
holder is found in the sentence, we just try to find the subject of
the sentence as the target candidate, because the author may be
the opinion holder and the target could be the subject of thesentence.
3) If still no target candidate is found in step 2), we just find theobject (VOB) in the sentence as the target because the object
could be the opinion target in case of no subject and no opinion
holder in the sentence.
Target candidate expansion is similar to holder candidate
expansion described in Section 4.3.2. If an opinion target is in thelist of opinion holder candidates, we remove it from the target list,
and then try to find another candidate.

5. RESULTS AND DISCUSSION
The above approaches were applied to the traditional Chinese task
at NTCIR-8 MOAT. Standard precision (P), recall (R) and Fmeasure (F) were used to evaluate the performance on eachsubtask. More descriptions about the gold standards and metrics
are covered in the overview paper [23]. All the results presented
in this section were released by the organizers.

5.1 Opinionated Sentence Recognition
Three runs were submitted for this task. The first run and the third
one are the same, only using the supervised lexicon-basedclassifier; the second run used the majority voting scheme basedon three classifiers: Maximum Entropy classifier, SVM classifier,and the supervised lexicon-based classifier. Table 2 reports theperformances of these three runs for this task.
Table 2. Results of opinionated sentence recognition
Run
1/3
2

Figure 2. Dependency Tree for Sentence e)
Furthermore, time nouns, numbers and words only containing one
Chinese character (except for pronouns) are removed from thecandidates, as they are unlikely to be opinion holders.

4.3 Identifying
OpinionOpinion-bearing Words

Targets

with

Here we use automatically identified opinion holders to help
opinion target identification. The procedure is as follows.
1) If a candidate of opinion holder is automatically identified
with a reporting verb in an opinionated sentence, we will try tofind the subject in the embedded clause as the target candidateby the following two steps:a) Find the subject of the object verb of the reporting verb.
For example, in sentence a) mentioned in section 2.2, we

P(%)
50.91
56.37

P(%)
91.97
85.71

R(%)
65.54
68.01

The voting scheme achieves the better performance on F-measurethan the supervised lexicon-based classifier only, by improving
the precision by 5.46%, but decreasing the recall by about 6.26%.
Overall, our system achieved the best F-measure on this task
amongst all the 14 submitted runs of the 7 participating teams,
and even the supervised lexicon-based classifier achieve the third
best F-measure among all runs. This result demonstrates theeffectiveness of the supervised lexicon-based classifier and theensemble method on the task of opinionated sentence recognition.

5.2 Opinion Polarity Classification
Three runs were submitted for this subtask, and they are all basedon the supervised lexicon-based polarity classifier. The third run
is the same with the first, and the difference between the first andthe second is that 1) the first is based on the opinionated sentencesin the first run for opinionated sentence recognition; 2) the second
is based on those opinionated sentences in the second run for
opinionated sentence recognition. Table 3 shows the results of the― 238 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

three runs of polarity classification for opinionated sentences inthe three runs, respectively.

features, and shallow parsing techniques would be further
investigated.

Table 3. Results of opinion polarity classification

Our proposed approach on identifying opinion holder/target ishighly dependent on the performance of dependency parsing andis not robust to the dependency errors, we would like to further
investigate machine learning approaches which can treat
dependency structures as one of the linguistic features and could
be more robust to dependency errors.

Run
1/3
2

P(%)
45.20
44.16

P(%)
41.95
38.52

R(%)
43.51
41.15

The supervised lexicon-based approach ranks the middle position
for opinion polarity classification. It still has much room toimprove.

5.3 Opinion Holder/Target Identification
Three runs were submitted for both opinion holder and target
identification based on the method introduced in Section 4, whichwere based on the three runs for opinionated sentence recognition,
respectively. The first two runs were based on the same method
without the following processing: 1) parentheses preprocessing insection 4.1; 2) the third factor holder in the previous sentence insection 4.2.1; 3) the second and third points in section 4.3. The
third one used all the processing and factors mentioned in section
4. The results are shown in Table 4.
Table 4. Scores for opinion holder/target identification
Run
1
2
3

HolderStrict Lenient
70.0
60.5
72.1
62.4
68.1
59.1

7. ACKNOWLEDGEMENTS
We acknowledge the help of HIT IR Lab (http://ir.hit.edu.cn/) for
sharing their Chinese LTP package including the Chinese
dependency parser.

8. REFERENCES[1] Steven Bethard, Hong Yu, Ashley Thornton, Vasileios
Hatzivassiloglou, and Dan Jurafsky. 2004. Automatic
Extraction of Opinion Propositions and their Holders, AAAI
Spring Symposium on Exploring Attitude and Affect in Text:
Theories and Applications.
[2] Yejin Choi, Claire Cardie, Ellen Riloff, and SiddharthPatwardhan. 2005. Identifying sources of opinions withconditional random fields and extraction patterns. In
Proceedings of HLT/EMNLP 2005. pp. 355-362.

TargetStrict Lenient
25.9
21.5
48.5
41.8
23.3
19.2

The second run achieved the second highest F-measure on theidentification of both opinion holders and targets among all
submissions, showing that the dependency parsing-based
approach on opinion holder and target identification is effective.
For opinion holder identification, the three runs ranked No. 3, 2
and 4 among the 8 runs submitted by 4 participating teams. For
opinion target identification, the second run ranked No. 2 among
the 6 runs submitted by 3 participating teams.
Please note the performance differences among the three runs
were not expected to be so large beyond our expectation because
the evaluation process of holder/target at Traditional Chinese side
was performed manually if the proposed answer was notconsistent with the gold answer. Therefore, there could be someinconsistencies due to annotators' different opinions, or errors.

6. CONCLUSION AND FUTURE WORKThis paper presents the approaches used in our system
participating in the opinion analysis task on traditional Chinese
texts at NTCIR-8. The system ranked No. 1 for opinionatedsentence recognition, No. 2 for identification of both opinion
holders and targets, the middle position for polarity classification.
It shows that 1) the combination of supervised lexicon-based
approach and machine learning techniques (namely, SVM andMaximum Entropy) is effective for opinionated sentence
recognition; 2) the dependency parsing-based approach on
opinion holder and target identification is effective despite its
simplicity.
There is much room for the system to improve on opinion polarity
classification. Hence part of our future work would be
investigating the unique features of opinion polarity classification.
The potential contribution of contextual information, topic-related[3] Hatzivassiloglou, Vasileios and Kathleen McKeown. 1997.
Predicting the Semantic Orientation of Adjectives.
Proceedings of ACL-97. 174-181.
[4] Minqing Hu and Bing Liu. 2004. Mining Opinion Features inCustomer Reviews. In Proceedings of the 19th National
Conference on Artificial Intelligence, pp. 755-760.
[5] Soo-Min Kim and Eduard Hovy. 2005. Identifying Opinion
Holders for Question Answering in Opinion Texts, In
Proceedings of AAAI-05 Workshop on Question Answeringin Restricted Domains. Pittsburgh, PA.
[6] Soo-Min Kim and Eduard Hovy. 2006. Extracting opinions,
opinion holders, and topics expressed in online news media
text, In Proc. of ACL Workshop on Sentiment andSubjectivity in Text, pp.1-8, Sydney, Australia.
[7] Youngho Kim, Seongchan Kim, and Sung-Hyon Myaeng.
2008. Extracting Topic-related Opinions and their Targets in
NTCIR-7, Proceedings of NTCIR-7 Workshop. Tokyo, Japan,
Dec. 16-19.
[8] Bin Lu. 2010. Identifying Opinion Holders and Targets with
Dependency Parser in Chinese News Texts. In Proceedingsof NAACL-HLT Research Student Workshop.
[9] Bin Lu, Benjamin K. Tsou and Oi Yee Kwong. 2008.
Supervised Approaches and Ensemble Techniques for
Chinese Opinion Analysis at NTCIR-7. Proceedings of
NTCIR-7. pp. 218-225. Tokyo, Japan.
[10] Lun-Wei Ku, I-Chien Liu, Chia-Ying Lee, Kuan-hua Chen
and Hsin-Hsi Chen. 2008. Sentence-Level Opinion Analysis
by CopeOpi in NTCIR-7. In Proceedings of NTCIR-7
Workshop. Tokyo, Japan.
[11] Marie-Catherine de Marneffe, Bill MacCartney andChristopher D. Manning. 2006. Generating Typed― 239 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Dependency Parses from Phrase Structure Parses. In LREC
2006.
[12] Jinshan Ma, Yu Zhang, Ting Liu and Sheng Li. 2004. A
statistical dependency parser of Chinese under small training
data. IJCNLP-04’s Workshop: Beyond shallow analyses Formalisms and statistical modeling for deep analyses.
[13] Xinfan Meng and Houfeng Wang. 2008. Detecting
Opinionated Sentences by Extracting Context Information. In
Proceedings of the Seventh NTCIR Workshop. Japan. 2008.
12.
[14] Zhou Ming and Huang Changning. 1994. Approach to theChinese Dependency Formalism For the Tagging of Corpus.
Journal of Chinese Information Processing.
[15] Jiaju Mei, Yiming Zhu, Yunqi Gao, Hongxiang Yin. 1996.
Tongyici Cilin (‫ٵ‬ᆠဲဲࣥ) (The 2nd Version). Shanghai
Lexicon Press.
[16] Bo Pang and Lillian Lee. 2008. Opinion mining andsentiment analysis, Foundations and Trends in Information
Retrieval, Now Publishers.
[17] Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002.
Thumbs up? Sentiment classification using machine learningtechniques. In Proceedings of EMNLP 2002, pp.79–86.
[18] Robi Polikar. 2006. Ensemble Based Systems in Decision
Making. IEEE Circuits and Systems Magazine, vol.6, no. 3,
pp. 21-45.
[19] Josef Ruppenhofer, Swapna Somasundaran, Janyce Wiebe.
2008. Finding the Sources and Targets of Subjective
Expressions. Proceedings of LREC 2008.
[20] Yohei Seki, David Kirk Evans, Lun-Wei Ku, Le Sun, HsinHis Chen, Noriko Kando, and Chin-Yew Lin. 2007.
Overview of Opinion Analysis Pilot Task at NTCIR-6. Proc.
of the Seventh NTCIR Workshop. Japan. 2007.6.
[21] Yohei Seki, David Kirk Evans, Lun-Wei Ku, Le Sun, HsinHis Chen, Noriko Kando. 2008. Overview of Multilingual
Opinion Analysis Task at NTCIR-7. Proc. of the Seventh
NTCIR Workshop. Japan. 2008. 12.

[22] Yohei Seki, Noriko Kando and Masaki Aono. 2009.
Multilingual opinion holder identification using author andauthority
viewpoints.
Information
Processing&
Management. Volume 45, Issue 2, March 2009, Pages 189199.
[23] Yohei Seki, Lun-Wei Ku, Le Sun, Hsin-His Chen, Noriko
Kando. 2008. Overview of Multilingual Opinion Analysis
Task at NTCIR-7. Proc. of the Seventh NTCIR Workshop.
Japan.
[24] Jilin Shi and Yinggui Zhu. 2006. The Lexicon of Chinese
Positive Words (ᓥᆠဲဲࠢ). Sichuan Lexicon Press.
[25] Veselin Stoyanov and Claire Cardie. 2008a. Annotating
Topics of Opinions. Proceedings of the Sixth International
Conference on Language Resources and Evaluation (LREC2008). Marrakech, Morocco.
[26] Veselin Stoyanov and Claire Cardie. 2008b. Topic
Identification for Fine-Grained Opinion Analysis.
Proceedings of the Conference on Computational Linguistics(COLING 2008).
[27] Peter D. Turney. 2002. Thumbs up or thumbs down?
Semantic orientation applied to unsupervised classificationof reviews, In Proceedings of ACL-02, Philadelphia,
Pennsylvania, 417-424.
[28] Xiaojun Wan. 2008. Using Bilingual Knowledge andEnsemble Techniques for Unsupervised Chinese Sentiment
Analysis. In Proceeding of EMNLP2008. pp. 553-561.
[29] Theresa Wilson and Wiebe Janyce. 2003. Annotating
opinions in the world press. Proceedings of the 4th ACL
SIGdial Workshop on Discourse and Dialogue (SIGdial-03).
[30] Ruifeng Xu, Kam-Fai Wong and Yunqing Xia. 2008.
Coarse-Fine Opinion Mining - WIA in NTCIR-7 MOAT
Task. In Proceeding of the Seventh NTCIR Workshop. Tokyo,
Japan, Dec. 16-19.
[31] Ling Yang and Yinggui Zhu. 2006. The Lexicon of Chinese
Negative Words (၈ᆠဲဲࠢ). Sichuan Lexicon Press.

― 240 ―