Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

PKUTM Experiments in NTCIR-8 MOAT Task
Chenfeng Wang*, Tengfei Ma*, Liqiang Guo,
Xiaojun Wan and Jianwu Yang
Institute of Computer Science & Technology, Peking University, Beijing 100871, China
Key Laboratory of Computational Linguistics (Peking University), MOE, China{ wangchenfeng,matengfei,guoliqiang,wanxiaojun, yjw}@icst.pku.edu.cn
ABSTRACT

Chinese side:This paper describes our work in the Simplified Chinese
opinion analysis tasks in NTCIR-8. In the task of detecting
opinioned sentences, various sentiment lexicons are used,
including opinion indicators, opinion operators, degree
adverbs and opinion words. The linear SVM model isselected as the main classifier, and four groups of features
are extracted according to punctuations, words andsentiment lexicons. We also try a two-step classification to
improve the SVM result. For extracting the opinion holder
and target, we use a synthesis of CRF and heuristic rules.
The evaluation results on NTCIR-8 MOAT Simplified
Chinese side show that our system achieves the best fmeasure in two tasks. This demonstrates that the proposed
framework is promising.

1. Opinionated subtask.
2. Opinion holder extraction.
3. Opinion target extraction.
We apply a machine learning method in opinionated sentences
identification task. The corpora in NTCIR-6, NTCIR-7 and
MPQA 1 are also used for training because the provided training
data in NTCIR-8 is not plenty enough. In addition, we translate
the Traditional Chinese corpus into Simplified Chinese and addthem into the training set.

Keywords: NTCIR, Sentiment Analysis,
Classification, Opinion Holder/Target Extraction

Subjectivity

1. INTRODUCTION
0B

Opinion mining, also called opinion analysis or sentiment analysis,
has become a hot area of computational linguistic. It focuses on
the analysis of subjectivity, sentiment and opinion extraction in
text. With the web’s fast development, opinion mining has more
and more applications, such as identifying the web users’
opinions toward products, people or events.
To date, a lot of work has been done for opinion mining. The task
becomes more mature and several subtasks are derived.
Subjectivity analysis, or opinionated identification, aims at
automatically recognizing subjective content. Sentimentclassification, or polarity classification, attempts to predict theorientation and strength of the sentiment towards the opinion
target in the subjective content. Besides, opinion extraction tries
to analyze further into sentence and extract opinion holders andopinion targets of the opinion expression.
Because many researchers perform their experiments on different
datasets, a fair comparison are difficult to make. NTCIR
Multilingual Opinion Analysis Tasks (MOAT) [8] proposes several
tasks to evaluate and compare different methods for opinion
mining.

FF

For opinion holder and target extraction, we use CRF [9] tocombine nouns or pronouns into candidate phrases and thenexploit a few heuristic syntactic rules to choose the best candidate.
Three runs are submitted to evaluate our classifier’s effectiveness.
The results show that our system achieves promising results on
two subtasks: the highest precision and F-measure on theopinionated task and the highest F-measure on the holder
extraction task among all submitted runs.
The rest of this paper is organized as follows. In Section 2 webriefly review the existing works on opinion mining. Our
opinionated identification system is described in Section 3, andSection 4 describes the method in holder and target extraction.
Section 5 gives the evaluation results and discussion. In the lastsection we give our conclusion.

2. Related Work
1B

Researches on opinion mining range from word-level to sentencelevel and document-level. In the early time, researches focus onwords [1]. Then, sentences and documents’ opinions are studied.
Mainly two kinds of approaches are proposed to solve theproblem. One is unsupervised[2][3], which uses linguisticknowledge on sentiment words and heuristic rules to predict thecontent’s opinion. The other is supervised [4], which extracts
features from labeled data and builds a classification or labelingmodel, and predicts the content’s opinion. The most commonlyused classification models include Naive Bayes[4], Maximum
Entropy[10] and Support Vector Machine[11]. Nowadays, more andmore methods combine the two approaches, by using thesentiment knowledge, heuristic rules as well as machine learningmodels [21].
Opinion holder/target identification is a more challenging
problem in the field of opinion analysis. Bethard[14] uses the

In NTCIR-8, we participate in three subtasks for the Simplified
1

Available at http://www.cs.pitt.edu/mpqa*The authors contribute equally to this paper.

― 228 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japantechnique of semantic parsing and syntactical features to extract
propositions and holders. Kim and Hovy[13] design a system toautomatically learn the syntactic features signaling opinion
holders using a Maximum Entropy ranking algorithm trained onhuman annotated data. Other researchers[7][9] use CRF [16] withsome extraction patterns.

Opinion Words
Opinion word lexicon is used in most of sentiment analysis
systems, and it plays a key role in opinionated sentences
identification. Our initial opinion word lexicon is collected fromthree linguistic resources:
1.

Our In-house Opinion Word List, which consists of 5975
words. Each word is marked with a positive or negative label
and its opinion degree. The Word List is first built based on
HowNet, then expanded by using Synonymy Thesaurus andthe SVM classifier, and manually check at last.

2.

The opinion word lexicon provided by National Taiwan
University (NTU), which consists of 2,812 positive words
and 8,276 negative words [17].

3.

The opinion word lexicon provided by Jun Li [18], which
includes 4468 negative words and 5567 positive words.

3. Detecting Opinionated Sentences
2B

In our system, we regard the sentence level subjective detection asa classification problem. First, data is processed by a POS (PartOf-Speech) Tagger and a NER (Named Entities Recognizing) tool.
Each sentence is represented by a vector of features extracted
from the dataset. After that, a Linear-SVM Classifier is applied to
get the basic classification result and finally an iterative classifier
is used to improve the prediction.

3.1 Dataset and Preprocessing
8B

NTCIR-6 and NTCIR-7’s corpora are used in our system. The
MPQA and our in-house labeled corpora are also used in lexicon
building and feature selection parts of our system. As we only
experiment in Simplified Chinese, all the Traditional Chinese
corpora are translated into Simplified Chinese using a translation
tool, ConvertZ2. While, the MPQA corpus is first processed andextracted in English, then the words are translated by Google
Translation Tool3.
Similar to most existing opinion mining tools, we first preprocess
the dataset before extracting the opinions of sentences. We use
our own word segmentation tool to get each Chinese word as wellas its Part-Of-Speech tag. After that, we use our in-house NER
Tool to recognize the named entities. Besides, we build somesentiment-related lexicons to identify the opinion features, which
are listed below.
Opinion Operators
An opinion operator lexicon is first collected from the sentimentdictionary of HowNet 4 and labeled operators in NTCIR-6, then
expanded using the Synonymy Thesaurus [19], and finally filteredusing labeled datasets. Only distinguishable verbs which appear
mostly in opinionated sentences are selected as good operators,e.g.㸼⼎(express), ໄ⿄(claim), Ⳍֵ(believe), ⲯ䌲(praise).
F

F

Opinion Indicators
An opinion indicator is a word indicating the orientation of an
opinion or the orientation trend of multiple opinions [20]. Wecollect 17 opinion indicators manually, such as Ԛᰃ(but), ᑊϨ(however), ሑㅵ(although).
Degree Adverbs
Degree adverbs, which frequently co-occur with opinion words,
can strengthen or weaken the degree of sentiment or even reverse
the polarity of the sentiment. Moreover, sometimes they can evenactivate a normal word to be sentimental. Some examples of thedegree adverbs include 䴲ᐌ(very), 㔎У(lack of), ϡ(not), ᇸ݊(especially), etc.
2

http://alf-li.tripod.com

3

http://translate.google.com

4

http://www.keenage.com
HU

The final lexicon consists of 28421 opinion words, such as ⃶୰(happy), ᔧ ༈ Ầ ୱ (a severe warning), ᗾ 㔾 (blame), 䲒 ෾(intolerable). We only use them as subject words without
considering whether they are positive or negative.
Strong Opinion Words
Only 4680 words’ probabilities of appearing in opinionatedsentences are no less than 50% in all the corpora. Some words
like ໻⇨(lordly or air), ᥼䖯(boost or push forward) might havedifferent meanings and sentiment in Chinese. So we decided tobuild a strong opinion word lexicon. We assume that all theidioms in the opinion word lexicon are more likely to be opinion
words and added them to the strong opinion lexicon. Besides, we
also add the word that appeared in opinionated sentences more
often than in objective sentences. Finally, we obtain a strong
opinion word lexicon with 6471 words.

3.2 Feature Selection
9B

Based on experiments on NTCIR-7’s corpus, we select fourgroups of features including punctuations features, words andentities features, sentiment features and collocation features.
Some of these features are borrowed from Ruifeng Xu’s work [5].
For each given sentence, we extract the following features andadd them to the vector space model.
Table 1. Features used in the opinionated subtask
Punctuations Features
Presence of quotation marks like “,ǋ,’, ǌand ā
Presence of colon followed by quotation marks
Percentage of punctuations in sentences
Words and Entities Features
The percentage of numeral words
The presence of pronoun
The presence of a named entity
The presence of a word which indicates a sequence
Lexical Subjective Clues
The presence of opinion operator
The presence of opinion indicator
The logarithm of percentage of opinion words
The logarithm of percentage of strong opinion words
The presence of degree verb

U

― 229 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan(OIF) use the presence of the word; the opinion words feature(OWF) and strong opinion words feature (SOWF) use thelogarithm of the percentage of the given words in sentence instead.
Tests on NTCIR-7 show that the logarithm of the percentage
performs better than term frequency or presence, which is also
proved on our self-labeled dataset.

Collocation Features
The presence of collocations between named entities andopinion operators
The presence of collocations between pronouns or nouns
and opinion operators
The presence of collocations between opinion operatorsand opinion words
The presence of collocations between pronouns andopinion words
The presence of collocations between nouns or pronouns
and opinion words
The presence of collocations between degree adverbs andopinion operators
The presence of collocations between degree adverbs andopinion words
The presence of collocations between nouns or namedentities and opinion words

3.2.4 Collocation Features
19B

Opinion is often expressed by several words together instead of
one word. By referring Rui’s paper, we make some changes on
careful tests and finally determine to use the eight features listed
in table 1:

With all these features our system achieves the F-measure valueof 0.702 under lenient evaluation and 0.743 under strict
evaluation on NTCIR-7’s Simplified Chinese test corpus.
Sentiment features and collocation features contribute the most,and the other two groups also make a slight improvement on some
corpora. The features different from other works are explained
below.

3.2.1 Punctuations Features

Combining the pronouns and nouns: The original features arecollocation between some words with pronouns or collocation
with some word with nouns respectively. We try to combine thefeatures and change them into collocation of some word withpronouns or nouns. Here the nouns include Named Entities. It canincrease the feature’s coverage with little lost in precision. After aseries of test, we find that the combined feature, collocations
between pronouns or nouns and opinion operators (PrPNOO)
contribute more than only use collocations between pronouns andopinion operators and collocation between nouns and opinion
operators.
The presence of collocations between opinion operators andopinion words (PrOpOw): This is an enhancement feature for
operator, and it would be more accurate.

16B

Punctuation features contain some special punctuation insentences, which may be good indicators for opinionated or unopinionated sentences. After browsing a lot of corpora and doing
experiments we find the following three useful features inopinionated judgment.
Colon followed by Quotation Marks (CQM): Although there isalready a quotation-related feature, we find that a sentence isusually someone’s words or some decelerations when it has aquotation strictly following a colon.
Punctuation Content (PuC): The percent of the punctuations
appearing in the sentence. The intuition behind this feature is that
we find that a sentence which has a low percentage ofpunctuations tends to be objective.

3.2.2 Words and Entities Features
17B

In the group of words and entities features, we add two features
which might be good indicators for objective sentence.
Percentage of numeral words (PeNW): Ratio between thenumber of punctuations and the number of all words andpunctuations. Numeral words often appear as data in objective
sentences.

3.3 Classification ModelBasic Classifier
10B

20B

We try several commonly used classifiers such as SVM, Naive
Bayes, Max Entropy and Decision Tree. They are all trained on a
combined dataset including NTCIR-6’s Traditional Chinese
corpus, NTCIR-7’s Traditional Chinese corpus and the training
set of NTCIR-7’s Simplified Chinese corpus. We use three
metrics for evaluation: Precision (P), Recall(R) and F-measure (F).
The results are shown in Figures 1 and 2.
SVM-linear uses the linear kernel of SVM and SVM-rbf uses the
Radial Basis Functions kernel of SVM. ME means Max Entropy
model. J48 is the Decision Tree Model using C4.5, and NB
is Naive Bayes. The linear SVM exceeds all the other models
under both lenient evaluation and strict evaluation. Thus wechoose the linear SVM as our basic classifier.
HH

3.3.2 Iterative Classifier
21B

Besides using the basic classifier, we also try some methods toimprove the basic classifier’s result. An iterative classifier isadopted according to Xu’s paper[6], and we only use theopinionated result other than polarities and relevance.

4. Extracting opinion holders and targets
3B

Presence of sequence indicator (PSI): The presence of a word
which can indicate the sentence to be a part of a sequence or list,
such as 佪‫(ܜ‬first of all), ㄀ϔ(first), ݊⃵(then), ϔ(1st, 1).

3.2.3 Lexical Subjective Clues
18B

This group of features is commonly used in most machinelearning based opinionated identification methods. In our system,
the opinion operator feature (OOF) and opinion indicator feature

Some former researchers consider the holder/target extraction as asequence tagging problem [7][9], and directly use CRF to address
the task based on some useful features, such as position, POS, anddependency tree. However, limited by the size of data sets and thescope that features can represent, labeling the holder always
results in a low recall. In this study, we use CRF to extract
candidate phrases, and then choose the best candidate as the final
holder/target with a few heuristic rules.

― 230 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

4.2 Finding Opinion Holders and Targets by
Heuristic Rules
12B

Figure 1.Comparison of different models under lenient
evaluation

Opinion holders are usually the subjects of the sentences and theyco-occurred with the opinion operators. Sometimes, however, theopinion holder may not exist in the operating sentence but isinherited from the neighboring sentences. Based on theobservation, we design some heuristic rules for identifying
opinion holders which are shown as follows.

Figure 2.Comparison of different models under strict
evaluation

1.

The holder is before an opinion operator (include a colon) or
following a quotes.

2.

If a candidate phrase is governed by a preposition, it is not
the opinion holder.

3.

If a sentence is in a quotation or contains only part of aquotation, the opinion holder must be before or behind thequotation (in previous or posterior sentences).

4.

If no candidate phrase conforms to the former conditions, try
to use nouns or pronouns as candidates.

5.

The opinion holder of subjective sentences, which have no
opinion operators or extract no candidate as a holder, isregarded as “԰㗙”(the author).

Opinion target identification is similar to the opinion holder
recognition. An opinion target usually is the object of an operator
or in the clause of opinion expression, but when the holder is inother sentences, it is the subject of the operating sentence. Due tothe property of our training set, the extracted phrases are mainly
subjects or objects of a verb. This further decreases thecomplexity of filtering rules.

5. Experimental Results and Discussion
4B

4.1 Generating Semantic Phrases
1B

Given a sequence x1x2…xn, we generate a corresponding sequenceof tags y1y2…yn. The possible tag values are chosen from the set{B, I, N}, where B indicates the beginning of a phrase, I is the
non-initial token of the phrase, and N represents that the word is
not in any phrases.
We use the “Chinese Proposition Bank 2.0” 5 as the original
training dataset, which is a corpus of Chinese text annotated withbasic semantic propositions. Due to the differences between ourtask and the task of semantic proposition identification, thelabeled information should be adapted. We only select the nounphrases and pronouns as the labeled phrases, because they arelikely to be the holder or target. The features for CRF mainly
include: word tokens, POS tokens, contextual features between
words and between POSs.
F

F

After training on the modified Chinese proposition bank, we
randomly choose a few samples from the file of “NTCIR8MOATSample_SC” and test the result to find possible wrong
patterns. Then the tested sentences are corrected and added into
the training set.

The NTCIR-8’s MOAT Simplified Chinese test corpus consists of
19 topics, 385 documents, including 4492 sentences and 4512
sentiment sub-sentences. There are two annotators and only thesentences labeled as opinionated by both annotators are
considered as opinionated. 18.99% of the sentences are
opinionated, and the others are objective.

5.1 Opinionated Sentence Recognition
13B

Three runs trained on different datasets were submitted. In run 1,
we used all corpora as training set, including NTCIR-6’s corpus,
NTCIR-7’s Simplified Chinese and Traditional Chinese corpus
and NTCIR-8’s Simplified Chinese training set. We applied boththe basic classifier and the iterative classifier for this run. The
NTCIR-7’s Simplified Chinese corpus and NTCIR-8’s Simplified
Chinese training set were used in run 2. Only the basic classifier
was applied. The NTCIR-7’s Traditional Chinese corpus wasadded to run2’s training set in run 3.
The results are measured by Precision(P), Recall(R) and Fmeasure(F). All evaluation results for identifying opinionatedsentences are given in Table 2.

5

http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=
LDC2008T07― 231 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

about the iterative classifier. Although the iterative classifier
improves run 1’ result, it does not make help on run 2’s result,and the result even decreases a little. Run 3 performs the worst,
but it is better than run 1 if run 1 just uses the basic classifier.
This proves that the run 3’s dataset is better than run 1 when
testing on NTCIR-8.

Table 2. The result for identifying opinionated sentences
Precision

RecallF-measureRun1

0.37210.83700.5152Run2

0.41340.83350.5527Run3

0.34050.90620.4950However, in contrast with NTCIR-7 Simplified Chinese’s results,
in which the values of precision and f-measure from the best run
achieved 0.5862 and 0.6839 under lenient evaluation, NTCIR-8’s
results are relatively low in both precision and F-measure. We
explain the phenomenon as follows.

Compared with other participants, our system achieves the highest
F-measure in run 2, which also achieves the best precision with alittle lose in Recall.

First of all, the test corpus in NTCIR8 may be harder to predict,and the differences in structure or semanteme between training
and test corpora often leads to worse results.

5.2 Opinion Holder and Target Identification
14B

In the opinion holder task, we achieves the best f-measure bothfor opinionated sentences and for all sentences among the four
participating groups and eight submitted results. Our target
identification also gains a good performance. This demonstrates
the effectiveness of our system. The results of the two tasks are
showed in Table 3 and Table 4 respectively, where the three runs
are based on the opinionated results of the above three runs in
Section 5.1, respectively.

Secondly, there may be differences in annotations. In NTCIR-8,
there are only two annotators and the lenient evaluation considers
all that sentence that be labeled as opinionated by both annotator
as opinionated. The percentage of opinionated sentences is only
18.99% while it is 38.32% in NTCIR-7.

Precision

RecallF-measure

Run1

0.892

0.736

0.806

At last, feature distributions are different. Table 5 shows thecoverage and precision of some well-performing features in
NTCIR-7, including CQM, OOF and PrPNOO. We can noticethat their precision declined a lot, which may be a reason why oursystem cannot get a high precision value.

Run2

0.896

0.732

0.805

Table 5. Features comparison between NTCIR-7

Run3

0.877

0.792

0.832

Run1

0.339

0.736

0.464



NTCIR-7

Run2

0.385

0.732

0.504



Coverage

Precision

Coverage

Precision

Run3

0.307

0.792

0.442

CQM

0.086

0.652

0.112

0.360

Table 4. Evaluations Results for Opinion Targets

OOF

0.315

0.681

0.391

0.407

PrPNOO

0.303

0.694

0.383

0.414

Table 3. Evaluations Results for Opinion Holders
Onlyfor
opinionate
d sentencesForallsentences

Precision
Onlyforopinionate
d sentencesForallsentences

Recalland NTCIR-8

F-measure

Run1

0.550

0.434

0.485

Run2

0.554

0.431

0.485

Run3

0.548

0.473

0.508

Run1

0.204

0.434

0.277

Run2

0.232

0.431

0.301

Run3

0.186

0.473

0.267

NTCIR-8

6. Conclusion
5B

5.3 Discussion and Data analysis

In this paper, we propose a series of feature selection methods
based on text information and opinion lexicon to solve theproblem of opinionated identification. In opinion holder andtarget extraction, CRF is used to extract candidate phrases ofnouns and pronouns. A few heuristic syntactic rules are adopted
to choose the best candidate. Our proposed system achieves a best
performance in the opinionated identification and holder
extraction tasks. However, the results still have large space to be
improved in further experiments by refining the model.

15B

Compared with other teams, our system achieves the best in Fmeasure and precision though the recall is not high enough in theopinionated sentence identification task. Looking through theresults of the three runs, we find the run 2 outperforms the others
and ranked the top in the list. It can be simply explained by thefact that NTCIR-7’s SC corpus is more similar with NTCIR-8’s
than other corpora. Furthermore, it might be the differences instyle between SC and TC when expressed. It is not always truethat the larger the training set is, the better the model would be.
For example the basic classifier’s result on Run 1’s train set isworse than on Run 2’s train set, thought the train set of Run 1 islarger than that of Run 2. We also perform some experiments

7. Acknowledgements
6B

This work was supported by NSFC (60873155, 60875033),
Beijing Nova Program (2008B03), NCET (NCET-08-0006),
National High-tech R&D Program (2008AA01Z421) andNational Development and Reform Commission High-tech
Program of China (2008-2441).

REFERENCES
7B― 232 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan[1] V. Hatzivassiloglou and K. R. McKeown. Predicting thesemantic orientation of adjectives. In Proceedings of the 35th
Annual Meeting of the Association for Computational
Linguistics, Madrid, Spain, 1997. ACL.

[13] Soo-Min Kim and Eduard Hovy. Identifying Opinion
Holders for Question Answering in Opinion Texts. In
Proceedings AAAI-05 workshop on Question Answering in
restricted domains. 2005.

[2] V. Hatzivassiloglou and J. M. Wiebe. Effects of adjectiveorientation and gradability on sentence subjectivity. In
Proceedings of the 18th International Conference on
Computational Linguistics, Saarbrucken, Germany, 2000.

[14] S. Bethard, H. Yu, A. Thornton, V. Hativassiloglou & D.
Jurafsky. Automatic extraction of opinion propositions andtheir holders. In Proceedings of AAAI Spring Symposium on
Exploring Attitude and Affect in Text.2004.

[3] Turney, P. 2002. Thumbs Up or Thumbs Down? Semantic
Orientation Applied to Unsupervised Classification ofReviews. Proceedings of the 40th Annual Meeting of the
ACL, Philadelphia, 417–424.

[15] Yejin Choi et al.. Identifying Sources of Opinions withconditional Random Fields and Extraction Patterns. In Proc
of EMNLP 2005.

[4] B. Pang, L. Lee, and S. Vaithyanathan, 2002.Thumbs up?
Sentiment classification using Machine Learning Techniques.
In Proceedings of the EMNLP conference.
[5] R.F.Xu,K.F. Wong and Y.Q.Xia, Coarse-Fine Opinion
Mining – WIA in NTCIR-7 MOAT Task. In NTCIR-7,2008.
[6] R.F, Xu, K.F. Wong, Q. Lu and Y.Q. Xia, Learning
Knowledge from Relevant Webpage for Opinion Analysis,
In Proc. IEEE/WIC/ACM WI-IAT, 2008.
[7] Kang Liu, Jun zhao. NLPR at Multilingual Opinion Analysis
Task in NTCIR. In NTCIR-7, 2008.
[8] Y. Seki, D.K.Evans, L.W. Ku, Overview of Multilingual
Opinion Analysis Task at NTCIR-7, In NTCIR-7,2008[9] Yejin Choi et al. Identifying Sources of Opinions withconditional Random Fields and Extraction Patterns. In Proc
of EMNLP 2005.
[10] B. Pang and L.L. Lee, A Sentiment Education: Sentiment
Analysis Using Subjectivity Summarization based on
Minimum Cuts. In ACL04, pp. 271-278, Spain, 2004[11] Riloff, J. Wiebe and T. Wilson, Learning Subjective Nouns
Using Extraction Pattern Bootstrapping. In CoNLL03, pp.2532, 2003.

[16] J. Lafferty, A. K. McCallum & F. Pereira. Conditional
Random Fields: Probabilistic Models for Segmenting andLabeling Sequence Data. In Proceedings of 18th
International Conference on Machine Learning. 2001.
[17] L.W. Ku and H.H. Chen, Mining Opinions from the Web:
Beyond Relevance Retrieval, Journal of American Society
for Information Science and Technology, pp. 1838-1850,
2007.
[18] Jun Li and Maosong Sun, Experimental Study on SentimentClassification of Chinese Review using Machine Learning
Techniques, in Proceeding of IEEE NLPKE 2007.
[19] J.J. Mei, Y.M. Zhu, Y.Q. Gao, H.X. Yin. Synonymy
Thesaurus (Second Edition). Shanghai LexicographicalPublishing House.
[20] L. W. Ku, T. H. Wu, L. Y. Lee, and H. H. Chen.
Construction of an evaluation corpus for opinion extraction.
in Proc. of NTCIR-5 Workshop. pp. 513–520, Tokyo, Japan,
2005.
[21] Yunqing Xia,Linlin Wang,Kam-Fai Wong and Mingxing
Xu.Sentiment Vector Space Model for Lyric-based Song
Sentiment Classification. ACL 2008.

[12] Y. Seki, L.W. Ku, L.Sun,H.-H. Chen, N. Kando, Overview
of Multilingual Opinion Analysis Task at NTCIR-8, In
NTCIR-8,2010.

― 233 ―