Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Opinion Analysis Task of Simplified Chinese
Monolingual at NTCIR-8
Yuan Kuang Bo Zhang Wenbin Pan Yujie Zhang Bin Zhang
Rongjun Li Yu Mao Yanquan Zhou
Research Center of Intelligence Science and Technology of Beijing University of Posts andTelecommunications, Beijing, 100876, China

zhouyanquan@bupt.cn
Abstract

language technology is of great value. For the Third

In this paper, we present our work for Simplified

Multilingual

Chinese Monolingual opinion analysis task at

NTCIR-8 workshop explore tasks including opinion

NTCIR-8 by BUPT. We participated in four of all

judgment (required), relevance judgment, answerness

tasks except opinion target detection and answerness

judgment, opinion holder extraction, opinion target

judgment, and submitted two runs for each task. For

detection and polarity judgment. The dataset about

opinion sentence detection, we propose some features

Simplified Chinese Monolingual coves many topics.

both semantic-level and grammar-level, and also

For more information about the tasks and dataset cansummarizes some syntactic structure templates tobe found in the overview paper [1].

achieve a more satisfactory classification results basedon TSVM. For opinion holder detection, we firstly use
CRF including six corresponding features to detect,then we propose two syntactic rules based on
opinionated trigger words from syntax trees taken as
additional features for the CRF to train our model. By
introducing

Statistical

Language

Modelswith

expansion of topic words we train a relevancejudgment model. To judge polarity, we compute thevalue of the text by our algorithm with a large-scaleemotional dictionary, and set a threshold, to classify
the sentiment polarity of sentence in each text.

OpinionAnalysis

Task(MOAT)At NTCIR-8, we participated in the four subtasks for
simplified Chinese monolingual, including opinion
and relevance judgment, opinion holder extraction andpolarity judgment. For identifying opinion sentences,
our training data comes from NTCIR-7 simplified
Chinese data and COAE (Chinese Opinion Analysis
Evaluation) that released forty thousand documents
consists of subjective and objective sentences, thenafter manually labeling, we extracted 2606 opinion
and 3496 non-opinion sentences as our training set.
We take the opinion judgment as binary classificationby using TSVM based on features both semantic-level

Key words:

and grammar-level and also propose some effective

NTCIR-8, Simplified Chinese Monolingual, opinion,

syntactic structure templates as opinion rules. For
opinion holder identification, we view it as sequential

relevance, opinion holder, polarity

labeling task via CRF, given a sentence of tokens, we

1. Introduction

classify the tokens into 3 categories, B(beginning ofNowadays, with the rapid development of thean opinion holder), I(inside an opinion holder) andinformation technology, a large number of users

O(outside an opinion holder). Then a chunk BI or B isexpress their views on the products in the forum, blog

thought as an opinion holder. So identifying opinion

and other platforms. On one hand, opinion analysis

holder via CRF is to label B, I, O for each token in acan provide us an effective way to help understandsentence. We firstly explore Conditional Random

and study sentiment, and on the other, it can be used

Field(CRF)

on Human-Computer interaction, question-answering

templates based on six features including contextual,system, watchdog to public opinion, personalized

opinionated trigger words, POS tags, named entity,

retrieval, etc. So opinion analysis based on naturaldependency that we adjust to be better helpful for― 265 ―

anddefinecorresponding

featureProceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

containing contextual dependency information andthe F-measure had further improve.

sentence structure feature that we propose. Then we

For identifying opinion holder, Yejin Choi and Claire

propose two syntactic rules with opinionated trigger

Cardie [6] adopted a hybrid approach that combined

words from the syntax trees and we take the rules as

Conditional Random Field (Lafferty et al.,2001) and aadditional features for the CRF combined with thevariation of AutoSlog(Riloff,1996a). In Soo-Min Kim

mentioned six features to extract opinion holder. For

and Eduard Hovy’s work [7][8], they firstly extracted

relevance judgment, we regarded it as retrieving

the syntactic paths between opinion holder candidates

sentences that are relevant to the given topic for eachand opinion_expressions then they treated paths as

document by introducing Statistical Language Models

features added to Maximum Entropy to train theirin the mean while expanding topic words and wemodel to select the most probable opinion holder;score each topic based on PMI. For polarity judgment,

their second work used FrameNet and Semantic Role

we set up a dynamic adjustment model for the

Labeling as an intermediate step to identify opinion

sentiment value of the words based on sentimentholder. In Yohei Seki’s work [9], his approach ofdictionary.

opinionThe rest of this paper is organized as follows: Section
2 gives an overview of the related works. Section 3
describes features we used for TSVM to judge
opinionated and propose some syntactic structure
templates. In Section 4, we show our proposed
syntactic rules combined with CRF to identify opinion
holder. And in Section 5 and Section 6 we give a brief
introduction for relevance and polarity judgment. The
evaluation results will be listed in Section 7. And
finally Section 8 concludes our paper with future

holderdiscrimination

between

was

based

on

the

authorandauthority

viewpoints in opinionated sentences, while Steven
Bethard [10] used an extension of semantic parsing
techniques coupled with additional lexical andsyntactic features, Ruifeng Xu [11] thought thenearest named entities before an opinion word as an
opinion_holder. Ruihong Huang [12] and Kang Liu[13] both used CRF model to identify opinion holder.
For relevance judgment, Evans [14] used standard
vector space model combined with TFIDF method to
calculate weight obtained good results. Li et al [15]works.

determined2.

extraction

Related Worktherelevancebycalculatingtheeigenvector of theme feature with inner product of the

For extracting opinion sentence as a relatively new

sentence, while Youngho Kim [16] used language

field is still at the exploratory stage and the current

model.

classification algorithm is still relatively simple. In

For polarity judgment, Bo Pang and Lillian Lee [17]contrast, research of English

carried out earlier.

used Naive Bayes, ME and SVM to classify sentences.

Wiebe et al. [2] proposed the following classification

Peter D. Turney [18] judged polarity based onfeatures: pronouns, adjectives, cardinal numeral,

unsupervised method. Soo-Min Kim and Eduard

modal verbs beside will, adverbs beside not, they also

Hovy[19] set up a sentimental classifier throughbelieve that the location of sentences in paragraphs

finding synonym based on WordNet.

also implies important information. Hatzivassiloglou,
Vasileios, and Wiebe [3] do further analysis on
characteristics of adjectives. They consider the
influence of dynamic adjectives, adjectives withsemantic information and graded adjectives to theclassification. Riloff and Wiebe [4] proposed thebootstrapping method based on a large number of
unlabeled corpus to identify opinion sentences in 2003.
By 2005, Wiebe and Riloff joined the extraction of theobjective models based on bootstrapping method and3.

Opinion sentence extraction

In my opinion, as long as the sentence contains atendentious opinion, whether it is declared by the first
person or quoted by the third person, is judged to be
an opinion sentence. In this task, we firstly use thesyntactic structure templates to tag sentences as
opinion sentence and then input the remaining data toa TSVM classifier. We think this task as binaryclassification using TSVM classifier to label the― 266 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

samples, each sentence is expressed in the form of

opinion or non-opinion. For example, there are moreeigenvectors, and weight of each dimensional

adjectives and adverbs in opinion sentences than incharacteristic is the number of times this characteristic

non-point sentences. Specifically, such as adjective +appears in this sentence. In addition, the opinion"de(ⱘ)", adverb + adjective and "de(ⱘ)" + noun,

training set is less than the non-opinion training set

are of more obvious opinions. "which resulted in imbalance of training data as section

In this experiment, we mainly take “N=1, N=2, N=3”

1 mentioned, so we train a TSVM classifier based onthese three cases into consideration. Extracting 1-POS,

strict result of NTCIR-7 to obtain more opinion

2-POS, 3-POS respectively from the training text,

sentences to balance the training set at first.

then sequencing them by calculating the CHI, to select3.1 Feature Selectionsome forward models of combination of parts ofReference the opinion and non-opinion sentences, we

speech as the characteristic. Formula of CHI is as

think that both of them imply their own uniquefollows: [5]characteristics on the semantic level and thegrammatical structure.
1)

Sentiment

N u(ADCB)2(AC)u(B D)u(A B)u(C  D)

(1)

The Formula calculates the CHI of the N-POS pattern

Sentiment means adjectives with some kind of
emotional tendencies. Usually, when someone says a
few words with emotional tendencies, we willconsider them as being of opinions.
2)

CHI( p, cj )p and the category c j . In this task, there are only two
categories, opinion sentence and non-opinion sentence.
A is the number of times p and c j , B is the number

Indicative verb

Indicative verbs are usually considered as signs ofexpressing words of views, such as "express","believe," "think," "forecast," "In my opinion," and soon. We finally summed up 64 verbs, which are used asof times p and c j , C is the number of times c j
without p , D is the number of times c j without p .

indicative verb vocabularies for experiment.

6)

3)

N-Word means a combination of sequence of N

Indicative adverb

N-WordIndicative adverbs, of which total number is 365, areconsecutive words. We think that a combination of amainly extracted from the experimental corpus. They

single word or several consecutive words also implies

include two kinds, one for adverbs of degree, such as

information of opinions, such as, "I" + "think","very( 䴲 ᐌ )", "most( ᳔ )", "somewhat( ⿡ ᖂ )","someone" + "indicates", ","+" however", "say" +"probably(໻ὖ)", "extremely(ᵕ݊)", "especially(Ḑ“colon” + “quotation”, are usually signs of opinion໪)", etc.; the other for adverbs pointing with mood or

sentences. We observe a unigram, bigram, tri-gram

attitude, such as "anyway( ড ℷ )", "rather( ᳾ ‫") ܡ‬,and then still sequence them by calculating the CHI"precisely(ᙄᙄ)", " Admittedly(೎✊) " and so on.

values to select some forward models of combination4)of consecutive words as the characteristic.

Interjection & Punctuation

Interjections, such as "Ah, La, Oh, Gosh, Well (“ଞ/3.2 Syntactic Structure Templates

ા/଺/ਔ/৻/ਸ਼”)", Punctuation, such as "?","!", they

By studying the large-scale data sets and the regular

are used when people usually express their own

opinion expression method, we find some kind of way

particular feelings, like praise, complaints, or doubt.

often used to express views, opinions, forecasts or5)comments. This kind of way is summarized as

N-POS

N-POS means a combination of sequence of N

following syntactic structure templates:

continuous parts of speech. We think that a

SS1

combination of a single part of speech or several//This template mains the subject of sentence is NP

continuous parts of speech, implies information ofand the predicate of sentence is one of indicative verb

― 267 ―

Subject NP  Predicate Indicative Verb  ... ADJP...

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

mentioned in section 3.1 and other child nodes which

Since the opinion holders are noun phrases, we use

have the same father node with the predicate verb

POS tags for each token as features including noun,contain adjective phrases.

verb, adverb, adjective, pronoun, punctuation, etc.

SS2

Subject NP  Predicate Indicative Verb  ... ADVP...

4)

Named entity features//This template is similar to SS1 but other child

The opinion holder may be a named entity, so we treat

nodes which have the same father node with thenamed entities as candidates of opinion holder. We

predicate verb should contain adverb phrases.

use the module of HIT_IR Lab language information

SS3

Subject NP  Predicate Indicative Verb  ...VCᰃ...

retrieval technology platform to gain named entity.

//The difference between SS3 and SS1 is that other5)

child nodes which have the same father node with the

For almost all opinion holders are triggered by

predicate verb contain ‘VC(ᰃ)’ instead of adjective

opinionated words, between them it must be thephrases.

dependency relationship. We use the module of
HIT_IR

4. Identifying opinion holder
4.1 Identifying opinion holder via
Conditional Random Field

sentence Y given the observation sentence X, not the
distribution of the following state given the current
state. CRF-based machine learning model can bearbitrarily added an effective feature vector to makefull use of contextual information, which canovercome the HMM independence assumptions andIn our work, we adopted Yet Another CRF toolkit
implemented by C++, whose nonlinear optimization
by

Limited-memory

With the definition of opinion holder that it must be
an entity and it must be preceded by an opinionatedtrigger word, we firstly explore the following features:must possibly contain contextual information of

6)

opinionated trigger words, so it’s necessary to analyze
the structure of each sentence. We obtain roughstructure of each sub-sentence, which is visualized bya triplet <before, center, after>.

4.3 Identifying opinion holder via
Syntactic Rules
and intuitively, there is a structural relationshipbetween an opinion holder and an opinionated trigger
word. So after analyzing parse trees we propose two

Parser, Figure 1 shows an example of a parsing
sentence.
Before setting the rules, we must locate an(VP(VV* in the parsing sentence. The two syntactic
rules are the followings:

opinion holder whose height is equal to that ofSince almost all opinion holders are triggered by

the node (VP or we treat the first node (NP that is

opinionated trigger words as is same to our definition,
words

that

are

labeledcontained in the node whose height is equivalentasto that of the node (VP.

<OPINION_OPR> from the corpus of NTCIR-6. We
treated opinionated trigger words features as binary to2)

If there is not a node (VP whose child is the node
(VP(VV*, then we treat the first node (NP as theindicate whether or not each token belongs to.
3)

If there is a node (VP whose child is the node
(VP(VV*, then we treat the first node (NP as theOpinionated trigger words features

all

Sentence structure features

The opinion holders are always depended on

1)

opinion holder.

extracted

dependency

opinionated trigger word(*) node signified byContextual features

For the opinion holder maybe a chunk BI or B, weweanalyze

extract opinion holder directly, using the Stanford

4.2 Feature Selection

2)

toretrieval

syntactic rules given the opinionated trigger words to

BFGS(LBFGS) algorithm.

1)

platform

information

On the basis of our definition about opinion holder

the inherent label bias of MEMM models.

trainedlanguage

relationship.

joint probability distribution of a particular label

are

Lab

technology

Conditional Random Field model is defined as theparameters

Dependency features

opinion holder whose height is equal to that ofPart-of-speech features― 268 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japanthe node (VP(VV* or we treat the first node (NPWe firstly find the relevant terms from Wikipedia. We

that is contained in the node whose height is

put the given topic into Wikipedia and find the terms

equivalent to that of the node (VP(VV*.

as the candidate expansion from the returned results.

As the syntactic rules indicate, in Figure1, there is a

Then compute PMI value between the topic andnode (VP whose child node is (VP(VV 䅸Ў(think ),candidate expansion as formula (3):then we regard the node (NP whose height is equal to

PMI

that of the node (VP that contains the node (NN ⾥ᄺᆊ(the scientists) as the opinion holder.

C (ext , top )C (ext ) C (top )(3)

Where top are the terms extract from the given

The syntactic rules take two features (SRFs) to the
CRF, which are showed in the following:

topic and ext are the expansion terms. C (ext , top ) ,SRF1: Whether or not each token is the child of theC (ext ) , C (top) are the number of the search
results from Google for searching ext & top ,located node (NP.

ext , top . The term with a higher value of PMI is

SRF2: if the frequency of the path from the locatednode (NP to opinionated trigger word node is only one,

more relevant to the topic. We sort the results
according to the PMI value and select 20 of them.

then we treat this path as null, and we also nullify the6.

tokens activated by this node (NP.

Polarity Judgment

As mentioned in section 1, we set up a model ofdynamic adjustment for the sentiment value of thewords based on sentiment dictionary (SVDA), as
Figure 2 shows.

Figure2. Dynamic adjustment model for the
Figure1. A parsing example.

5.

sentiment value of the words.
In the model of SVDA, we record the times of theRelevance sentence Judgment

For the sentence relevance subtask we applied

sentiment word that occurs in the training data. T pos

language model method. For the given topic, weis the times of the word in dictionary occur in theevaluate the relevance between the topic and thepositive train data, and Tneg is in the negative train
data. Then we calculate the Sentiment _ val

sentence as formula (2),(thesentiment value of the word which can be found in thep (Q / d )¦ p(Q / Q ', d ) p(Q '/ d ) | ¦ p(Q / d ) p(Q '/ d ) (2)
Q '

Q '

dictionary) by the following formulas:

Q is for the terms extract from the given topic andval

Tpos  Tneg / Tpos  Tneg

(4)

Q ' is for the expansion terms. The key point of thismodel is query expansion so we expand the query as

If ( val > 0)follows:

Sentiment _ val
If ( val < 0)

― 269 ―

Tpos u val(5)

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Sentiment _ val

Tneg u val

effectiveness of the features we used for CRF and(6)

syntactic rules we proposed, as well as the additional

If ( val = 0 && word is in positive dictionary)Sentiment _ val 1

features that are extracted from the syntactic rules.
However, there still exists some problems, like in one(7)sentence there may have two opinion who put forward
Otherwise:

Sentiment _ val

an opinion, it is hard for us to identify; another1

problem we need to research is how we should limit(8)

the length of opinion holder, some results are wrong
So each word in the sentiment dictionary will be given
a sentiment value, and we accumulate the value ofeach sentiment word in the sentence, and set the result

since the opinion holder may only contain one word
after segmentation or may contain more than one
word.

as the sentiment value of the sentence, denoted as
Table 4 displays the results of relevance judgment,

Sentence _ val .

If the Sentence _ val is greater than a threshold, thesentence will be classified as positive, otherwise, if
the Sentence _ val is less than another threshold, theand shows the effectiveness of expansion terms given
topics, the problem is in both runs we the sentences
we proposed are not enough, we should focus on
improving the recall effectively.

sentence will be classified as negative, others will be
Table1. The lenient results of opinion sentence

charged as neutral.

7.

judgment

Experiments and Evaluation

As mentioned in section 1, we submitted two runs for

Lenient

Precision

Recalleach task. The lenient evaluation results for simplified

Run1

38.43

67.53

48.98

Chinese monolingual are listed in the following

Run2

35.02

87.81

50.07

Tables. Table 1 shows the results of opinion sentence

Table2. The results for identifying opinion holder

judgment, our system ranked second among all

for opinionated sentences

participants, while the F-value of run2 is a little higher

Precision

RecallF-value

F-valuethan run1, the reason is that in run1 we only use

Run1

0.929

0.473

0.627

TSVM to classify the testing set, but in run 2 we

Run2

0.894

0.496

0.638

firstly use the syntactic structure templates to tag

Table3. The results for identifying opinion holder

sentences as opinion sentence and then we input thefor all sentences

remaining testing data to the TSVM classifier. So we
Precision

obtain a higher recall with a little lower precision.
Also, the results indicate the features we extract for
TSVM are effective for judging. In the following
experiments including opinion holder identification,

F-value

Run1

0.31

0.473

0.375

Run2

0.242

0.496

0.325

Table4. The lenient results of relevance sentence
judgment

polarity and relevance judgment, we firstly filter
testing set by using our opinion sentence extraction

RecallLenient

Precision

RecallF-valuesystem, so the recall of all of these three subtasks is a

Run1

98.23

39.64

56.49

little lower, which can not completely indicate the

Run2

97.9

51.55

67.55

performance of our systems. Though, we still got
good performance.

Table5. The lenient results of polarity judgment
Lenient

Precision

RecallF-value

Table 2, 3 shows the results of opinion holder

Run1

60.41

27.18

37.49

identification for opinionated sentences and for all

Run2

58.13

31.13

40.55

sentences submitted, as can be seen, the precision of
the system is good and stable, which demonstrates― 270 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

8.

Conclusion and Future works

This paper shows the methods for Multilingual
Opinion Analysis Task (MOAT) by BUPT to
Simplified Chinese monolingual, and submitted fourtasks

including

opinionated,

opinionholderidentification, relevance and polarity judgments. For
opinionated judgment, our results ranked two, whichindicates the effectiveness of the syntactic structure
templates we proposed and the TSVM classifier basedon features both semantic-level and grammar-level.
To extract opinion holder, we propose a novel
syntactic rules to identify opinion holder combinedwith CRF. As to judge relevance, we considered the
expansion of terms based on Statistical Language
Models. For polarity judgment, we set up a dynamic
adjustment model for the sentiment value of the words
based on sentiment dictionary.

9.

Acknowledgments

This work is supported by the research center ofIntelligence Science and Technology of Beijing
University of Posts and Telecommunications.

10. Reference[1] Y.Seki, D.Evans, L. Ku, H.Chen, N. Kando, andC.Lin. Overview of Opinion Analysis Pilot Task
at NTCIR-8. In Proceedings of the 8th NTCIR
Workshop Meeting, Japan, 2010.
[2] Janyce Wiebe M, Bruce, Rebecca F, O'Hara,
Thomas P. Development and use of a gold
standard data set for subjectivity classifications. In
Proc. 37th Annual Meeting of the Assoc. for
Computational Linguistics (ACL-99). Association
for Computational Linguistics, University of
Maryland, June, pp. 246-253.
[3] Hatzivassiloglou, Vasileios, Janyce Wiebe.
Effects of Adjective Orientation and Gradabilityon
Sentence
Subjectivity.
International
Conference on Computational Linguistics(COLING-2000).
[4] Ellen Riloff, Janyce Wiebe. Learning Extraction
Patterns for Subjective Expressions. Conferenceon Empirical Methods in Natural Language
Processing (EMNLP-03).ACL SIGDAT. Pages
105-112.
[5] Yinming Yang, Jan O.Pedersen. A Comparative
Study on Feature Selection in Text Categorization.
Proceedings of the 14th International Conference
on Machine Learing. Nashville ˖ Morgan
Kaufmann, 1997˖412̚420.
[6] Yejin Choi , Claire Cardie , Ellen Riloff ,Siddharth Patwardhan, Identifying sources of
opinions with conditional random fields andextraction patterns, Proceedings of the conferenceon Human Language Technology and Empirical
Methods in Natural Language Processing,

pp.355-362, October 06-08, 2005, Vancouver,
British Columbia, Canada.
[7] Kim, Soo-Min and Eduard Hovy. 2005.
Identifying Opinion Holders for Question
Answering in Opinion Texts. Proceedings of
AAAI-05 Workshop on Question Answering in
Restricted Domains.
[8] S.-M. Kim and Eduard Hovy. Extracting Opinions,
Opinion Holders, and Topics Expressed in Online
News Media Text. In Proceedings of workshop on
sentiment and subjectivity in text at proceedingsof the 21st international conference on
computational linguistics/the 44th annual meetingof the association for computational linguistics(COLING/ACL 2006) (pp. 1-8). Sydney,
Australia.
[9] Y. Seki, “Opinion holder extraction from author
and authority viewpoints”, Proceedings of the
30th annual international ACM SIGIR conference
on Research and development in information
retrieval (SIGIR 2007), pp. 841-842.
[10] Bethard, Steven, Hong Yu, Ashley Thornton,
Vasileios Hatzivassiloglou, and Dan Jurafsky.
2004. Automatic Extraction of Opinion
Propositions and their Holders, AAAI Spring
Symposium on Exploring Attitude and Affect in
Text: Theories and Applications.
[11] Ruifeng Xu, Kam-Fai Wong and Yunqing Xia.
Opinmine-Opinion Analysis Sytem by CUHK for
NTCIR-6 Pilot Task. In Proc of the six NTCIR
Workshop. May.2007, Japan. 2007[12] Ruihong Huang, Longxin Pan and Le Sun.
ISCAS in Opinion Analysis Pilot Task:
Experiment with sentimental dictionary basedclassifier and CRF model. In Proc of the six
NTCIR Workshop. May.2007, Japan. 2007[13] K. Liu, and J. Zhao, “NLPR at Multilingual
Opinion Analysis Task in NTCIR7,” pp. 226-231.
[14] D.Evans. A low-resources approach to Opinion
Analysis: Machine Learning and Simple
Approaches. In Proceedings of the 6th NTCIR
Workshop Meeting, Japan, 2007.
[15] Y.Li, K. Bontcheva and H. Cunningham.
Experiments of Opinion Analysis on the Corpora
MPQA and NTCIR-6. In Proceedings of the 6th
NTCIR Workshop Meeting, Japan, 2007.
[16] [14] Youngho Kim, Seongchan Kim,
Sung-Hyon Myaeng. Extracting Topic-related
Opinions and their Targets in NTCIR-7. In
Proceedings of the 7th NTCIR Workshop Meeting,Japan,2008.
[17] Pang, Bo, Lillian Lee, and Shivakumar
Vaithyanathan. 2002. Thumbs up? SentimentClassification
using
MachineLearning
Techniques, Proceedings of EMNLP-2002.
[18] Turney, Peter D. 2002. Thumbs up or
thumbsdown? Semantic orientation applied tounsupervisedclassificationofreviews,
Proceedingsof
ACL-02,
Philadelphia,
Pennsylvania,417-424.
[19] Kim, Soo-Min and Eduard Hovy. 2004.
Determining the Sentiment of Opinions.
Proceedings of COLING-04. pp. 1367-1373.
Geneva, Switzerland.

― 271 ―