Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Detecting Opinions and their Opinion Targets in NTCIR-8
Yoonjung Choi, Seongchan Kim, Sung-Hyon Myaeng
Korea Advanced Institute of Science and Technology
335 Gwahak-ro Yuseong-gu Daejeon, South Korea+82-423507753{choiyj35, sckim, myaeng}@kaist.ac.kr
ABSTRACT
Identifying an opinion target, a primary object of the opinion
expression (e.g., the real-world object, event, and abstract entity),is helpful for extracting target-related opinions and detecting userinterests. This paper presents a novel framework for target-based
opinion analysis, which extracts opinionated sentences andidentifies their opinion targets from news articles. To determine
whether a sentence includes opinions, we utilize opinion lexicons(i.e., predefined clue words) and linguistic patterns. In identifying
the opinion target, candidates are generated and examined forexistence of four different features. We attempt to capture therelationship between an object target and opinion clues and utilize
a document theme. For evaluation, we used English news articles
from New York Times, provided by NTCIR-8 MOAT andannotated opinionated sentences and theirs opinion targets.
Experimental results show that our proposed method is promisingalthough many additional issues remain to be studied in the future.

Categories and Subject Descriptors
H.3.3 [Information Storage and Retrieval]: Information Searchand Retrieval – Information Filtering. I.2.7 [Artificial
Intelligence]: Natural Language Processing – Text Analysis.

General Terms
Algorithms, Experimentation.

Keywords
Opinion Analysis, Sentiment Analysis, Opinion Judgment,
Opinion Target Identification, Multilingual Opinion Analysis
Task (MOAT), NTCIR.

1. INTRODUCTION
Deliberating on some problems such as social issues andindividual problems or deciding on something such as problems
and products, one of the important information is what others arethinking about the same issues [1]. Nowadays, as online opinion
resources, such as online news, blogs, online communities,

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,requires prior specific permission and/or a fee.
Conference’10, Month 1–2, 2010, City, State, Country.
Copyright 2010 ACM 1-58113-000-0/00/0010…$10.00.

reviews, and forums, grow up, Internet users can easily search forothers opinion (or public opinion) about their interest. In fact,
31% of the whole Americans were online for gathering
information and exchanging views via email during the 2006
election campaign season. Also, about 14 million people
contributed to political discussion and activity through reading
and writing Web articles [5].
Opinion analysis is to recognize opinions and their related
attributes such as polarity (positive or negative), opinion holder,and opinion target [3]. These opinions and related attributes canbe of service to not only individuals but also governments andcompanies [6]. So, many researchers have been interested inopinion mining from several data such as news articles, blogs andproduct reviews, and several methods which obtained reasonable
performances have been developed [7,8,9]. However, the most ofthem have focused on identifying subjective or objective
sentences, judging polarity values, and extracting opinion holders,
not identifying opinion targets.
Opinion data, such as news articles, blogs, and reviews, contain atleast one topic and its opinions. Therefore, people are practicallyinterested in opinions relevant to a particular topic, not any
opinions. However, almost every previous study has neglected a
relevant topic when opinions were decided. Moreover, opinions
which contain interesting topics (e.g., popular products and social
issues) can be more closed to user interest. For instance, many
people are interested in popular movies, political election, andsensational events which obtain many and various opinions fromthe public [3]. This topic can be defined as the real-world object,
event, or abstract entity, which would be the primary subject of
the opinion as intended by the opinion holder [10]. As previous
studies [3,11] referred such a topic as an opinion target, in thispaper, we adopt the same concept.
The opinion target identification would be very useful for several
applications such as extracting target-related (or topic-related)
opinions, estimating trends (e.g., popular products and issues) anddetecting user interests. However, there have been a few researchworks on the opinion target identification [3,8]. Kim et al. [3]attempted to build an opinion target classifier using syntactic pathinformation between opinion clues and an opinion target andsyntactic dependency feature (i.e., syntactic patterns such as“Verb opinion clues + Target” and “Adjective opinion clues +Target”). In another work [8], they used FrameNet semantic rolelabeling to identify an opinion target.
Although these syntactic features and semantic role labeling makea great contribution to identify an opinion target, they cannot
solve completely this problem [11]. With this question in opinion
target identification, we are motivated to participate in the 3rd― 249 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Multilingual Opinion Analysis Task (MOAT) in NTCIR which
has five subtasks: opinionated sentence judgment, relevancesentence judgment, polarity judgment, opinion holder
identification, and opinion target identification [2]. Among them,
we joined in 2 subtasks: opinionated sentence judgment andopinion holder identification.
In this paper, we focus on the opinion target identification. To
judge opinionated sentences, we utilize opinion lexicons, such as“good”, “criticize”, and “death”, and devised linguistic patterns.
As lexicons, a sentiment lexicon and a list of appraisal verbs areused. In the opinion target identification, unlike previous works[3,8] applied syntactic paths, syntactic dependencies and semanticroles as features, in this paper, we attempt to detect an opinion
target by considering sentence and document information andcollocation information between an opinion target and opinion
clue words, not syntactic features. We first extract all nounphrases as candidates and build a classifier which determines
whether a candidate is suitable for an opinion target or not.
Features used for the classifier include the degree to which it iscorrelated with the document-level theme, collocation information
between the candidate and opinion clues, and the opinion score of
the candidate.
The rest of our paper is organized as follows. In Section 2, we
explain several related works. Section 3 presents our proposed
methods, and Section 4 shows and briefly discusses the evaluation
results. Finally, we conclude in Section 5.

2. Related WorkIn this paper, we focus two tasks: opinionated sentence judgment
and opinion target identification. There have been a few previous
works about opinion target identification, so we also investigate
opinion holder identification because they may have similar
approaches.

2.1 Opinionated Sentence Judgment
Most previous works have adopted statistical classification
methods such as Naïve Bayes and Support Vector Machine with
lexical features [3]. Early work by Wiebe et al. [12] developed theprobabilistic classifier to automatically discriminate the subjective
and objective category. The subjective sentence refers to aspectsof language used to express opinions. They utilized the Navie
Bayes classifier with several features: the presence of a pronoun,an adjective, a cardinal number, a modal other than “will”, and an
adverb other than “not”, whether the sentence begins a newparagraph, and the co-occurrence of words and punctuation marks.
Hatzivassiloglou and Wiebe [13] studied the benefit of dynamic
adjectives (oriented adjectives) and gradable adjectives for thesentence-level subjectivity classification. Yu et al. [14] studied
separating opinions from facts at the document-level andsentence-level on TREC 1 8, 9, and 11 collections. They also
applied the Naïve Bayes and multiple Naïve Bayes classifier, andthe presence of semantically oriented words, the average semanticorientation score of the words, and the N-grams are used forfeatures.
Riloff and Wiebe [15] suggested bootstrapping methods for the
subjectivity classifier. From the labeled data, they generated
1

patterns to represent subjective expressions, and these patterns
were utilized to identify more subjective sentences. Then, basedon these patterns, they classified subjective sentences. In [16],they developed the learning method for the rule-based subjectivity
classifier which looks for subjective clues. Stepinski and Mittal[17] also developed the new sentence classification using a
Passive-Aggressive algorithm trained on unigram, bigram, andtrigram features. Also, the recent work [18] considered relations
between word sense disambiguation and subjectivity [1].

2.2 Opinion Target Identification
The opinion target identification task newly proposed in NTCIR-7[22]. There had only one team which participated in this task.
Kim et al. [3] proposed the statistical classifier to detect an
opinion target. Syntactic paths between a candidate and opinion
clues (e.g., “NP VP VBZ”), syntactic dependencies based on thepart-of-speech of opinion clue words, the presence of topic words,
and the named entity type are used for features.
Kim and Hovy [8] attempted to extract an opinion topic (or target)
utilizing FrameNet semantic role labeling and manually mapping
semantic roles to an opinion topic in news articles. They firstly
identified frame elements in a sentence and detect an opinion
topic from these. The mapping table to map frame elements to an
opinion topic was manually built by them.

2.3 Opinion Holder Identification
Kim and Hovy [8] applied FrameNet semantic role labeling todetect an opinion holder. The method was same with the opinion
topic identification (we mentioned in Section 2.2), and it had thedifferent mapping table for the opinion holder extraction. Anotherpaper [19] suggested the new approach which used syntactic paths
between an opinion holder’s candidate and the expression(opinion clue words). They used the Maximum Entropy ranking
model to obtain the scores and to rank candidates. Kim et al. [4]used the dependency tree to identify an opinion holder. They
extracted the nominal subject of the verb as the opinion holder inan opinionated sentence.
Choi et al. [20] proposed the hybrid approach which used both
Conditional Random Fields (CRFs) and extraction patterns.
Features for CRFs are lexical and syntactic information andsemantic words. Seki et al. [21] suggested the new opinion holder
identification method based on a differentiation between theauthor and authority viewpoints in opinionated sentences. They
exploited the named entity and grammatical information and used
NTCIR-6 data for evaluation.

3. Opinion Mining System
Our proposed system consists of two parts. The first one is for theopinionated sentence judgment which decides whether a given
sentence expresses an opinion or not (i.e., subjectivity orobjectivity). The other part is for the opinion target identification
which detects opinion targets within opinionated sentences.

3.1 Opinionated Sentence Judgment
For the goal of this work, classifying a sentence into opinionatedor factual (i.e., non-opinionated), we made two assumptions:x Opinionated sentences contain opinion clue words
x Some patterns may appear in opinionated sentences

TREC, http://trec.nist.gov/.

― 250 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Based on these assumptions and commonly used methods in therelated works, we address our opinionated sentence judgment
approach which combines two methods: pattern matching method
and opinion scoring method.
For opinion patterns, first, we analyzed NTCIR-8 sample data [2]to devise several patterns to determine the subjectivity of asentence. In this paper, the pattern means several lexical clues,
such as “believe”, which strongly reveal one’s opinion in a
sentence. We observed some verbs and auxiliary verbs such as“would” could be strong evidence for the non-factual statement.
We finally select 15 opinion patterns through the analysis ofsample data (Table 1). These patterns are used to detect the
subjectivity.
Table 1. Opinion patterns frequently shown in sample data
Main Verb“believe”, “insist”, “claim”, “criticize”,“think”, “advise”

AuxiliaryVerb“would”, “could”, “may”, “should”,“might”, “will”

Phrase“in fact”, “unfortunately”, “consequently”

For other sentences which cannot discover opinion patterns, weemploy opinion lexical resources such as SentiWordNet 2 [23],which has 5,321 sentiment words, and Appraisal verbs 3 [24],which has 703 appraisal verbs or adjectives. SentiWordNet is afreely available lexical resource in which each synset of WordNet
is associated to two numerical scores, such as positive score andnegative score, representing how much the term in the synset has
positive value and negative value, respectively. Using these twoscores, we get the subjectivity score which can be calculated asfollows:

SubjScr (t )OpiScr (t )OpiScr ( S )SentiWordNet, http://sentiwordnet.isti.cnr.it.
Appraisal
http://lingcog.iit.edu/arc/appraisal_lexicon_2007b.tar.gz.

tS

If a sentence is matched to the opinion patterns, we decide thegiven sentence is opinionated. Also we determine a correct answer
for subjectivity that if the opinion scores of the given sentence isover the certain threshold. We optimize the threshold value using
sample data.

3.2 Opinion Target Identification
In previous works, syntactic and lexical features or devised
syntactic patterns are utilized to identify an opinion target.
Furthermore, we can think an opinion target may employ similarfeatures to an opinion holder, and many researchers have also
used syntactic and lexical information to detect an opinion holder.
However, those are mostly sentence-level features. In this paper,
we show the document-level feature and collocation between an
opinion target and opinion clue words (corpus-level feature) also
are useful.
First, we should decide which parts in a sentence are candidates
for an opinion target. As we mentioned above, the opinion target
means the real-world object, event, or abstract entity. Therefore,
the unit of an opinion target can be confined to be a noun phrase.
So, for opinion target’s candidates, we extract all atomic nounphrases from each sentence using a parser4.
For opinion target identification, we have three intuitions:x Document-level theme can be an opinion target
x An opinion target may contain opinion clue words
x An opinion target will frequently co-occur with several
opinion clue words

where t is each term, and PosScr(t) and NegScr(t) is a positivescore and negative score of a term t, respectively. In this case,SentiWordNet has different scores depended on senses, so weapply the maximum score among them [4]. Appraisal lexicon ishand-picked appraisal words from Levin’s Verb Classes [25] andadjectives extracted from the NTCIR corpus. However, Appraisal
lexicon is just the word set and has no appraisal score. Then, to
combine with SentiWordNet, we give an appraisal score tomembers of Appraisal Verb set. The appraisal score is 0.3, whichis empirically set, since a major portion of opinion clue words in
SentiWordNet have more than 0.3 as a sentiment score while nonopinion clues or trivial words have less than 0.3. The opinion
score of a term is computed to combine the objectivity score andappraisal score, and the opinion score of a sentence is calculated
to sum up all opinion scores of the terms in a sentence as follows:3¦ OpiScr (t )where S is a sentence and AppScr(t) is an appraisal score of aterm t.

PosScr (t )  NegScr (t )2

SubjScr (t )  AppScr (t )Verbs,

Based on these, we generate a statistical classifier which decides
whether a given candidate is appropriate for an opinion target ornot. We consider the four types of features, and they are explained
below. In order to incorporate these features, we adopt the linear
regression method which works well with a small size of training
data (the number of documents in sample data is only 12).
Our first intuition is an opinion target should be related to a
document-level theme. Most sentences in a document talk about adocument theme (or topic). So, if a sentence expresses an opinion(e.g., positive or negative), it may be close connection between
the target of opinion and the document theme. For this, we exploit
two features. One is whether or not the given candidate’s keyword
appears on the title section in a document since most document
topics are mentioned on the title section in a document. For
example, in a document with the title “Bohemian Border Town
Blues: The Euro’s to Blame!” and the sentence “Things will get
worse, not better, when his country adopts the euro ...”, “theeuro” can be not only a document theme in this document but also
an opinion target in this sentence. The other feature is the4

We used Stanfordparser.shtml).

― 251 ―

Parser(http://nlp.stanford.edu/software/lex-Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

document-level language model. We adopt the unigram language
model:

P( NP | D)P(t1 ,..., tn | D)

 P(t | D)ni 1

4. Experimentiwhere NP is a candidate of an opinion target, D is a document, t isa term, and n is the number of terms in a candidate NP.
Furthermore, the probabilities are estimated as follows:

P (ti | D)c(ti , D)¦tD

From the analysis of sample data that the topic is “Bali Island
Terrorist Bombing” [2], we discover several opinion targets,
which have strong opinions, comprehend opinion clue words suchas “bomb”, “threat”, “terrorism”, “attack”, and so on. Also,
according to the definition, several events (e.g., social issues, caraccidents, and big festivals) can be an opinion target, andsometimes the name of events contains opinion clue words suchas “festival”, “accident”, “strike”, etc. So, the opinion score ofterms in a candidate are also deliberated.

¦ OpiScr (t )n

The NTCIR-8 MOAT collection [2] consists of 21 topics (150
articles). They are New York Times articles which had been
published in 2002-2005. Among them, one topic containing 12
articles is provided as the sample data, and others (20 topics – 138
articles) are used for test (i.e., formal running data). Each sentence
in articles was annotated about opinionatedness, topic-relevance,
polarity, opinion holder, and opinion target. We utilize only the
sample data (one topic) for training and others (20 topics) for
testing.

4.2 Experimental Result and Discussion
First, we present the experimental result with the sample data.
Based on this experiment, we optimized our system and submitted
our result with the formal running data. The result of formal
running data will be described in Section 4.2.2.

i

i 1

n

4.2.1 Sample Data

where OpiScr(t) is the opinion score of term t which is estimatedas follows:

OpiScr (t )In this section, we present the effectiveness of our proposed
method using NTCIR-8 MOAT data which will be explained in
Section 4.1. Precision, recall, and F-measure are used to evaluate
opinionated sentence judgment and opinion target identification
method, which are suggested in [2].

4.1 NTCIR-8 MOAT Data Collectionc(t , D)

where c(t, D) is the number of occurrences of a term t in adocument D.

OS ( NP)and collocation information between an opinion target andopinion clues.

Max[Scr (t | POS ), Scr (t | NEG)]

We got two annotation results (i.e., two annotators participated in
the annotation task) about each articles. So, two different goldstandard are existed. One is a lenient standard that an answer isagreed by at least one of two annotators, and the other is a strict
standard that an answer is agreed by all two annotators.

where Scr(w|POS) and Scr(w|NEG) are positive and negative
sentiment score of the term t, respectively, which are provided by
SentiWordNet. The range of score is 0 d Scr (.) d 1 .
The last intuition is an opinion target will co-occur with various
opinion clues in several documents (or corpus). The opinion
target detected one sentence can also be detected in othersentences. Moreover, opinion targets usually are constructed withmanifold opinion clues while others (non opinion targets) are
written with a few opinion words. For example, the opinion target“SARS” is usually used with opinion clues such as “threat”,“worry”, and “warn” in several sentences even though the non
opinion target “the network structure” isn’t. Therefore, we utilize
collocation information between a candidate and opinion clues asfollows:

C ( NP, OW )| S NP  SOW || S NP | u | SOW |where OW is the opinion clue set (we use SentiWordNet), SNP and
SOW is the set of sentences which contain NP and one of opinion
clues OW, respectively, and |.| is the number of sentences in a set.
Overall, the feature space to identify an opinion target consists offour types: whether the title section contains an opinion target,
document-level language model, opinion target’s opinion score,

Table 2. Opinion judgment result in sample data
Features

Precision

RecallF-measureOpinion Scoring(OS)

0.15250.12440.1370Opinion Rules(OR)

0.38260.30110.3369OS + OR

0.4625(0.3612)0.3559(0.2827)

0.4022(0.3172)

Table 2 is opinionated sentence judgment result in sample data.
Strict cases represent in parentheses. Performance on the opinion
judgment on sample data was best when opinion patterns andopinion scoring method are applied together whereas using only
opinion patterns or opinion scoring method could not cover many
opinion sentences.
Table 3 shows the result of opinion target identification in sample
data. We regard as a correct answer when a detected opinion
target is exactly matched with the gold-standard. Document-level
language model and collocation information between an opinion
target and opinion clues are the most important for the opinion
target identification while a feature about the appearance on thetitle section is not. The opinion score feature is also non-trivial
since the effect of using only this feature is not trifling. When― 252 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

those features are combined, we can know all features contributeto identify the opinion target even if the difference is not
significant.

opinion targets do not occur very often or sometimes are notconnected with opinion clue words.
Table 4. Formal running result

Table 3. Opinion target identification result in sample data
Features

Precision

RecallF-measure

Title Section(TS)

0.10400.07740.0887

Opinion Score(OS)

0.26670.33330.2963

Language Model(LM)

0.31510.44640.3695

Collocation(CO)

0.29720.44050.3549LM + CO

0.32310.50000.3925LM + CO + OS

0.32690.50600.3972LM + CO +OS + TS

0.3282(0.2744)0.5060(0.5844)0.3981(0.3734)

Task

Precision

RecallF-measureOpinion
Extraction

0.18880.6526

0.2943

Opinion Target
Identification

0.231

0.346

0.277

5. Conclusion

4.2.2 Formal Running Data
The result of the formal run is shown in Table 4. Comparing withsample data results, the overall performance is deteriorated
because our submitted result is over-fitted to the sample data. The
sample data contained only one topic while test collection
contained 20 topics. Therefore, the sample data would notembrace all features of opinionated sentences and opinion targets,so we could miss something important.
In the opinion extraction task, our proposed system has higher
recall value than NTCIR-7 result [22]. Since we utilized abundant
opinion words (e.g., SentiWordNet and Appraisal Verbs), oursystem detected a lots sentences. Among them, some sentences are
extracted because they contain plenty of weak opinion clue words.
Also, SentiWordNet had different sentiment score according tosenses, but we didn’t deliberate on this case and considered only
the maximum score about each word. Besides, several nonopinion words also have a sentiment score in SentiWordNet.
Therefore, while the precision value is low, the recall is high.
The opinion target identification task has overall better
performance than previous work [22]. Then, we can verify the
document-level information and collocation information are also
useful to detect an opinion target. However, the precision andrecall values are still low, partly due to the strict matching
criterion. Most notably, inability to handle anaphora gave many
incorrect opinion targets. This anaphora can be found very often
in news articles. Also, since the gold-standard is made by humanannotators, it contains many pronouns (e.g., “it”, “them”, and“him”) as opinion targets. Synonymy problem is also significant.
Some opinion targets could occur in other sentences with different
surface-level, but these regard as different opinion targets.
Another weakness is associated with reliance on the term
frequency and the presence of opinion clues in the features. Some

This paper presents two tasks, opinionated sentence judgment andopinion target identification, which are subtasks in NTCIR-8
MOAT. For the opinion judgment at the sentence-level, theopinion score and opinion patterns are used. If a sentence ismatched to opinion patterns or contains strong or several opinion
clue words, it would be detected as an opinionated sentence. For
the opinion target identification, we extract noun phrases ascandidates and build a classifier, which determines whether acandidate is suitable for an opinion target or not, with four typesof features: the appearance on the title section, document-level
language model, opinion score of a candidate, and collocation
information between a candidate and opinion clues. Although we
didn’t obtain good performance, we show not only sentence
information (e.g., syntactic and lexical features), but also
document information and collocation information are helpful toidentify an opinion target.
As future works, we should investigate the opinion target
boundary detection although we confined it to a noun phrase inthis paper. The anaphor and synonym problem are also important
issue in natural language processing since almost texts have suchproblem. Moreover, after analyzing plenty of opinion documents,
to discover significant features or characters of the opinion target
should be necessary for improvement.

6. ACKNOWLEDGMENTS
This work was financially supported by the MKE (The Ministryof Knowledge Economy), Korea, under the ITRC (Information
Technology Research Center) support program supervised by the
NIPA (National IT Industry Promotion Agency (NIPA-2010C1090-1011-0008)).

7. REFERENCES[1] Bo Pang and Lillian Lee. 2008. Opinion Mining and
Sentiment Analysis. Foundations and Trends in Information
Retrieval, Vol. 2, Issue 1-2.
[2] Yohei Seki, Lun-Wei Ku, Le Sun, Hsin-Hei Chen, andNoriko Kando. 2010. Overview of Multilingual Opinion
Analysis Task at NTCIR-8. In Proceedings of the 8th NTCIR
Workshop Meeting on Evaluation of Information AccessTechnologies, NII, Japan.
[3] Youngho Kim, Seongchan Kim, and Sung-Hyon Myaeng.
2008. Extracting Topic-related Opinions and their Targets in
NTCIR-7. In Proceedings of the 7th NTCIR Workshop― 253 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, JapanIn Proceedings of the Conference on Empirical Methods in
Natural Language Processing (EMNLP).

Meeting on Evaluation of Information Access Technologies,
pages 247-254, NII, Japan, December 2008.
[4] Jungi Kim, Hun-Young Jung, Sang-Hyeob Nam, Yeha Lee,and Jong-Hyeok Lee. 2008. English Opinion Analysis for
NTCIR7 at POSTECH. In Proceedings of the 7th NTCIR
Workshop Meeting on Evaluation of Information AccessTechnologies, pages 241-246, NII, Japan, December 2008.
[5] Lee Rainie and John Horrigan. 2007. Election 2006 Online:
The Number of Americans Citing the Internet as the Sourceof Most of their Political News and Information Doubled
since the Last Midterm Election. Pew Internet & American
Life Project Report, January.
[6] Yoonjung Choi, Yuchul Jung, and Sung-Hyon Myaeng.
2010. Identifying Controversial Issues and their Sub-topics
in News Articles. In Proceedings of Pacific Asia Workshop
on Intelligence and Security Informatics. Accepted.

[15] Ellen Riloff and Janyce Wiebe. 2003. Learning Extraction
Patterns for Subjective Expression. In Proceedings of the2003 Conference on Empirical Methods in Natural Language
Processing (EMNLP).
[16] Janyce WEibe and Ellen Riloff. 2005. Creating Subjective
and Objective Sentence Classifiers from Unannotated Texts.
In Proceedings of the 7th World Congress on Intelligent
Control and Automation, China.
[17] Adam Stepinski and Vibhu Mittal. 2007. A Fact/Opinion
Classifier for News Articles. In Proceedings of the 30th
Annual International ACM SIGIR Conference on Researchand Development in Information Retrieval, pages 807-808,
ACM, Amsterdam, The Netherlands.

[7] LiZhuang, Feng Jing, and Xiao-Yan Zhu. 2006. Movie
Review Mining and Summarization. In Proceedings of the
15th ACM International Conference on Information andKnowledge Management (CIKM), pages 44-50, ACM, USA.

[18] Janyce Wiebe and Rada Mihalcea. 2006. Word Sense andSubjectivity. In Proceedings of the Joint Conference of theInternational Committee on Computational Linguistics andthe Association for Computational Linguistics (COLINGACL).

[8] Soo-Min Kim and Eduard Hovy. 2006. Extracting Opinion,
Opinion Holder, and Topics Expressed Online News Media
Text. In Proceedings of the Workshop on Sentiment andSubjectivity in Text, pages 1-8, Association for
Computational Linguistics, Sydney.

[19] Soo-Min Kim and Eduard Hovy. 2006. Identifying andAnalyzing Judgment Opinions. In Proceedings of the Main
Conference on Human Language Technology Conference of
the North American Chapter of the Association ofComputational Linguistics, pages 200-207, New York.

[9] Namrata Godbole, Manjunath Srinivasaiah, and Steven
Skiena. 2007. Large-Scale Sentiment Analysis for News andBlogs. In Proceedings of the International Conference on
Weblogs and Social Media (ICWSM), Colorado, USA.

[20] Yejin Choi, Claire Cardie, Ellen Riloff, and SiddharthPatwardhan. 2005. Identifying Sources of Opinions with
Conditional Random Fields and Extraction Patterns. In
Proceedings of the Human Language Technology
Conference / Conference on Empirical Methods in Natural
Language Processing (HLP-EMNLP).

[10] Veselin Stoyanow and Claire Cardie. 2008. Annotation
Topics of Opinion. In Proceedings of the 6th International
Language Resources and Evaluation (LREC), pages 32133217, European Language Resources Association, Morocco.
[11] Josef Ruppenhofer, Swapna Somasundaran, and Janyce
Wiebe. 2008. Finding the Sources and Targets of Subjective
Expressions. In Proceedings of the 6th International
Language Resources and Evaluation (LREC), Marrakech,
Morocco.
[12] Janyce M. Wiebet, Rebecca F. Bruce, and Thomas P. O’Hara.
1999. Development and Use of a Gold-Standard Data Set for
Subjectivity Classifications. In Proceedings of the 37th
Annual Meeting of the Association for Computational
Linguistics (ACL).
[13] Vasileious Hatzivassiloglou and Janyce M. Wiebe. 2000.
Effects of Adjective Orientation and Gradability on Sentence
Subjectivity. In Proceedings of the In 18th International
Conference on Computational Linguistics (COLING), pages
299-305.
[14] Hong Yu and Vasileious Hatzivassiloglou. 2003. Towards
Answering Opinion Questions: Separating Facts from
Opinions and Identifying the Polarity of Opinion Sentences.

[21] Yohei Seki, Noriko Kando, and Masaki Aono. 2009.
Multilingual Opinion Holder Identification using Author andAuthority Viewpoints. Information Processing andManagement 45, pages 189-199.
[22] Yohei Seki. 2007. Overview of NTCIR-7 MOAT. In
Proceedings of the 7th NTCIR Workshop Meeting on
Evaluation of Information Access Technologies, NII, Japan,
December 2008.
[23] Andrea Esuli and FabrizioSebastiani. 2006. SentiWordNet:
A Publicly Available Lexical Resource for Opinion Mining.
In Proceedings of the 5th Conference on Language
Resources and Evaluation (LERC’06), pages 417–422,
Geneva.
[24] Casey Whitelaw, NavenduGarg, and ShlomoArgamon. 2005.
Using Appraisal Groups for Sentiment Analysis. In
Proceedings of the 14th ACM Intermational Conference on
Information and Knowlwdge.
[25] Beth levin. 1993. English Verb Classes and Alternations: apreliminary investigation. University of Chicago Press,
Chicago and London.

― 254 ―