Extraction of Opinion Sentences using Machine Learning: Hiroshima City
University at TCIR-7 MOAT
Daisuke Kobayashi

Hidetsugu Nanba Toshiyuki Takezawa

Hiroshima City University
3-4-1 Ozukahigashi, Hiroshima 731-3194 JAPAN

Phone: +81-82-830-1584
kobayashi@nlp.its.hiroshima-cu.ac.jp, {nanba, takezawa}@hiroshima-cu.ac.jp

Abstract
We propose a machine learning-based method
for extracting opinion sentences using 13 features
including about 760,000 of sentence-final expressions. We submitted two systems to the Japanese
Subtask of the MOAT at 'TCIR-7 Workshop, andobtained F-values of 0.5615 and 0.3319 using lenient gold standard, and 0.5213 and 0.3561 using
strict gold standard, respectively.
Keywords: Sentence-final expressions, Machinelearning, Opinion Extraction.

two kinds of lists of sentence-final expressions. One isa list of expressions that appeared frequently in 585
opinion sentences frequently, and the other is a list of
expressions that appeared in 2,167 non-opinion sentences. Both opinion and non-opinion sentences were
provided by the organizers of NTCIR-6. In our work,
we prepared several lists of sentence-final expressions,which were created from three corpora: "TSC4","NTCIR-6", and "News". In the following, we describe the detail of the lists from these corpora.

2.1.1
Lists Produced from TSC4 andTCIR-6 Corpora

We used machine learning to build our system. In
this section, we describe several features used for machine learning.

The "TSC4" corpus comprises 12,000 Japanese
sentences from the databases of Mainichi, Yomiuri,
Asahi, and Nikkei newspapers written in 2003 and2004. Human assessors annotated opinion tags to each
opinion sentence in the corpus. Using the corpus, wecreated a list of sentence-final expressions by extracting uni-gram, bi-gram, tri-gram, and 4-gram of sentence-final expressions from opinion sentences. Finally, we obtained about 14,000 expressions.
We also prepared two other kinds of lists from"NTCIR-6" corpus [5]. "NTCIR-6" is a corpus distributed at the opinion analysis pilot task in the
NTCIR-6 workshop. Our lists were created by extracting uni-gram, bi-gram, tri-gram, and 4-gram ofsentence-final expressions from opinion and nonopinion sentences in the corpus. Finally, we obtained
142 expressions for opinion, and 198 for non-opinion,
respectively.
The number of expressions from NTCIR-6 is muchsmaller than that from TSC4. We created both lists
manually. We spent a couple of months on creationfrom the "TSC4" corpus, but we could spend only a
few weeks on the "NTCIR-6" corpus.

2.1

2.1.2

1

Introduction

Recently, opinion extraction and sentiment analysis
have been considered as a major research topics in the
NLP community. We have studied automatic extraction of opinion sentences using the TSC4 corpus,
which was created for the purpose of opinion summarization studies [1]. We used machine learning for
opinion extraction, and obtained precision of 0.8382
and recall of 0.8184. To confirm the effectiveness of
our method using another data set, we participated in
the MOAT task at NTCIR-7 [2].
The remainder of this paper is organized as follows.
Section 2 explains how to extract trend informationfrom newspaper articles. Section 3 reports on the experiments, and discuss the results. We present someconclusions in Section 4.

2

Opinion Extraction

Sentence-final Expressions

For the extraction of opinion sentences, we focusedon sentence-final expressions, because useful clues,
such as modality, tense, and aspect, appear there. In
the past NTCIR workshop, many groups used sentence-final expressions for opinion extraction [3] [4].
Kanamaru et al. [3] applied machine learning method
for opinion extraction. They used words and uni-gramto 10-gram strings at the end of sentences as the features for machine learning. Mizuguchi et al. [4] also
focused on sentence-final expressions. They prepared

Lists Produced from ews Corpus

In addition to the lists described in Section 2.1.1,
we prepared two sentence-final expressions lists using"News" corpus. Generally, objective facts are mentioned in most newspaper articles, while authors' subjective opinions are described in editorials. We therefore collected sentence-final expressions from bothnewspaper articles and editorials in "News" corpus
using the following procedure.
1.

Collect newspaper articles on the front pageand editorials from News corpus. As a News

2.
3.
4.

corpus, we used Mainichi newspapers published in the 14 years between 1993 and 2006.
Extract N-grams of sentence-final expressions
from newspaper articles and editorials.
Rank the N-grams from both newspaper articles and editorials using cost criteria1 [6].
Re-rank both N-grams using the following
equation.


New score of an sentence-final expressions in editorials = score of the expression in editorials – score of the expression in newspaper articles
New score of an sentence-final expressions in newspaper articles = score of theexpression in newspaper articles – scoreof the expression in editorials

Here, step four degraded sentence-final expressions
that appear in both newspaper articles and editorials
frequently. Finally, we obtained 351,914 sentencefinal expressions for opinion (from editorials), and402,054 for non-opinion (from newspaper articles).
We summarize the number of sentence-final expressions obtained from each corpus in Table 1.




2.3

The number of parentheses: Whether a sentence contains statements of organizations or
person or not.
Frequencies of each part of speeches in a sentence: Mainly used for detecting non-opinion
sentences containing only symbols or particularmarks.

Utilization of Features in Surrounding
Sentences

From the results of our pilot study, we found thatshort sentences tend to obtain low precision, because
the information contained in such sentences are insufficient. We therefore used features in the surrounding
sentences as well as the features in each target sentence5.

3

Experiments

3.1

Submission

In addition to sentence-final expressions, we used thefollowing features for machine learning.

To confirm the effectiveness of our approach, we
submitted two systems: "HCU-1" and "HCU-2" to theformal run of the Japanese Subtask at NTCIR-7
MOAT [2]. Both systems used the "NTCIR-6" corpus
and a sample data provided by the organizers of the
MOAT in advance of the formal run, as training data.
Each sentence in the data was annotated by three assessors. If at least two assessors identified a sentence
as opinion, the HCU-1 system regarded it as a positive example, while the HCU-2 required all assessors
to identify a sentence as opinion for it to be considered positive.



3.2

Table 1. The number of sentence-final expressions
TSC4
TCIR-6
ews
opinion
14,000142351,914
non-opinion
198
402,054

2.2






Other Features

Sentence position: The position of each sentence in articles or editorials. The values arenormalized between 0.0 and 1.0 by dividing by
the number of sentences in the article.
Sentence length: The number of characters andwords (morphemes)2 in each sentence.
Subjects: A subject in a sentence.
Subject types: Organization, person's name, or
location3.
Tense: Past, present, or future4.

We used the data provided by MOAT organizers
for evaluation. The data consists of 18 topics including 249 documents in total. All the submitted systems
were evaluated in terms of precision, recall, and Fmeasure using two gold standards: lenient (two of
three assessors have the same annotation) and strict(all three assessors must have the same annotation)
[2].

3.3
1

Simple term-frequency method tends to rank shorter
terms higher. To solve this problem, the cost criteria
ranks terms using the following equation.
Score(term) = frequency * length of the term

Data and Evaluation

Results and Discussions

The experimental results are shown in Table 2. The
results showed that HCU-1 obtained higher precisions
than HCU-2 in both lenient and strict evaluations.
However, the number of opinion sentences extracted
by HCU-1 is much smaller than HCU-2.

2

To count the number of words in a sentence, weused a Japanese morphological analyzer MeCab.
(http://mecab.sourceforge.net/)
3
To identify the subject types, we used a JapaneseparserCaboCha(http://chasen.org/~taku/software/cabocha/).
Theparser can be used as a named entity recognizer.
4
To identify the tense of each sentence, we used aJapanese parser KNP (http://www-lab25.kuee.kyotou.ac.jp/nl-resource/knp.html).

5

We used YamCha software package, which specializes in text chunking based on TinySVM software(http://chasen.org/~taku/software/yamcha/). We useda polynomial kernel of degree 2, and used a value of 9
for the window size.

Lenient
Strict3.3.1

Table 2. Experimental result
P
R
HCU-1
0.61900.5138HCU-2
0.77540.2111HCU-1
0.48940.5577HCU-2
0.65440.2446

F
0.56150.33190.52130.3561

Typical Errors of Our System

The followings are typical errors of our systems.
(1) The lack of sentence-final expressions (40%)(2) Over-extraction of opinion sentences (20%)(3) Titles and headlines (15%)(4) Others (25%)
In the following, we describe each of these errors.
(1) The lack of sentence-final expressions (40%)
Although we prepared many of sentence-final expressions, some opinion sentences could not be extracted because they lacked sentence-final expressions.
One such case follows.
Table 3. Example of expressions that are not contained in our lists
ot contained expresSimilar expressions insions
our lists
否めない (cannot deny)
否定できない (cannot
deny)
非難する (blame)
責める (blame)
We compared these expressions with our lists, andfound that similar expressions are contained in ourlists. Recently, several methods to detect synonyms orsimilar expressions have been proposed, and theireffectiveness was confirmed in various applications,such as query expansion for information retrieval [7]and evaluation in text summarization [8]. The low
recall by our methods might be improved applying
these methods to our lists.
(2) Over-extraction of opinion sentences (20%)
Following three examples were identified as nonopinion sentences by human assessors, while our systems extracted them as opinion sentences, because
sentence-final expressions of these sentences are contained in our lists.
[origintal]
同じ気持ちを抱いているに違いない。
[translation](They must feel the same way.)[orignial]「わが国は事務総長提案に賛同できない[translation]("Our country cannot agree with the proposal by adirector-of-the-executive-office".)[orignial]
構造改革を進めるべきだ。
[translation](We should promote our structural reform.)(3) Titles and headlines (15%)
Some assessors annotated some titles or headlines
as opinion, but our systems could not extract them,because cue expressions do not appear at the end ofthese sentences. The following are two examples thatassessors judged as opinion sentences.
[original]◇納得できない-川田龍平さん[translation](◇I cannot be convincing. – Mr. Ryuhei Kawada)[orignial]◇政府と協調必要--白川浩道・ＵＢＳウォーバーグ
証券チーフエコノミスト[translation](◇ Cooperation with the government is necessary –Hiromichi Shirakawa, a chief economist of UBS
Warburg)

3.3.2

Error Analysis for each Topic

Evaluation results for each topic are shown in Table 4. Topics N05 (Kosovo civil war), N12 (El Nino),
and N14 (greenhouse gas) gave low F-values. All ofthese topics tend to contain many statements or conversational sentences in comparison with other topics.
Generally, opinion sentences are written in the present
tense, but these sentences are often written in the pasttense, and our systems could not identify them as
opinion sentences. We show an example as follows.
[original]
ＮＯＡＡのトム・カール博士は「１９００年以降、陸地の気温の上昇率は海上より２０％も高く、最近の高温傾向は人間の活動による温室効果ガスの増加と関係していると言える」と指摘した。
[translation](Dr. Tom Carl of NOAA pointed out "Since 1900, thepercentage of rise of land temperature is more than
20% higher than that of sea temperature. It can be said
that this phenomenon in these few years relates to theincrease of greenhouse gas owing to the human activity.")

3.3.3

Effects of Sentence-final Expressions

To confirm the effects of each list of sentence-final
expressions, we removed one of these lists, conductedmachine learning, and calculated recall and precision
values. When the list from "TSC4" corpus was removed, the recall score fell by about 2 %, while theprecision score was almost the same. When the list
from "NTCIR-6" corpus was removed, the precisionand recall scores fell by 5% and 2%, respectively.
When the list from the "News" corpus was removed,
both precision and recall scores fell by about 10%.
From these results, we can conclude that the"News" corpus is the most effective among three lists.
The list from the "TSC4" corpus contributes to improve recall. Although the size of the list from the

"NTCIR-6" corpus is very small, it significantly improved the precision.

4

Conclusions

We have proposed a machine learning-based
method for extracting opinion sentences using about
760,000 of sentence-final expressions and some otherfeatures. We submitted two systems to the Japanese
Subtask of the MOAT at NTCIR-7 Workshop. We
obtained F-values of 0.5615 and 0.3319 using lenient
gold standard, and 0.5213 and 0.3561 using strict gold
standard, respectively.

5

Acknowledgements

The authors would like to express their gratitude to
the organizers of the MOAT.

References[1] M. Okumura, T. Hirao, and H. Nanba. TSC4: A Corpus
for Opinion Summarization, and A Workshop using the
Corpus. In Proceedings of the 11th Annual Meeting of the
Association for Natural Language Processing, 2005. (inJapanese)[2] Y. Seki, D. K. Evans, L.-W. Ku, L. Sun, H.-H. Chen,
and N. Kando. Overview of Multilingual Opinion Analysis Task at NTCIR-7. In Proceedings of the 7th NTCIR

Workshop Meeting on Evaluation of Information AccessTechnologies: Information Retrieval, Question Answering and Cross-lingual Information Access, 2008.
[3] T. Kanamaru, M. Murata, and H. Isahara. Japanese Opinion Extraction System for Japanese Newspapers Using
Machine-Learning Method. in Proceedings of the 6th
NTCIR Workshop Meeting, pp.301-307, 2007.
[4] H. Mizuguchi, M. Tsuchida, and D. Kusui. Three-Phase
Opinion Analysis System at NTCIR-6. In Proceedings of
the 6th NTCIR Workshop Meeting, pp.330-335, 2007.
[5] Y. Seki, D. K. Evans, L.-W. Ku, H.-H. Chen, and N.
Kando. Overview of Opinion Analysis Pilot Task at
NTCIR-6. In Proceedings of the 6th NTCIR Workshop
Meeting, pp.265-278, 2007.
[6] K. Kita, Y. Kato, T. Omoto, and Y. Yano. A Comparative Study of Automatic Extraction of Collocations from
Corpora: Mutual Information vs. Cost Criteria. Journal of
Natural Language Processing, Vol.1, No.1, pp.21-33,
1994.
[7] H. Nanba. Query Expansion using an AutomaticallyConstructed Thesaurus. In Proceedings of the 6th NTCIR
Workshop, pp.414-419, 2007.
[8] L. Zhou, C.-Y. Lin, D.S. Munteanu, and E. Hovy. ParaEval: Using Paraphrases to Evaluate Summaries Automatically. In Proceedings of the 2006 Human Language
Technology Conference of the North American Chapterof the Association for Computational Linguistics (HLTNAACL), pp.447-454, 2006.

recall

Table 4. Evaluation results for each topic
F-value
TOPIC

Topic ID

precision

N03

0.58090.5267N04

0.61830.6807N05

0.18920.35000.2456

コソボの民族紛争(Kosovo civil war)

N06

0.37140.61900.4643

ネパール(Incident with Nepal’s ruling family (royalty))

N07

0.39680.64100.4902

中国系インドネシア人に対する暴動(Attacks toward Chinese Indonesian people)

N08

0.5686

0.67440.6170

米国対マイクロソフト(Lawsuit American Government against Microsoft)N09

0.31250.48780.3810

核実験(Nuclear weapons tests)

N10

0.37100.57500.4510

シリアと中東和平プロセス(Suriyah in the Middle East Peace Process.)N110.4706

0.42110.4445

AOL とネットスケープ(The relationship between AOL and Netscape)N120.38000.27540.3194

エルニーニョ
(El Nino)N13

0.38460.60000.4687

中国とロシア(The relationship between China and Russia)

N140.31580.24000.2727

温室効果ガス(Greenhouse gasses)

N15

0.54290.59380.5672

NATO とポーランド(The relationship between NATO and Poland)

N16

0.62110.60820.6146

タイとアジア経済危機(Thailand in the Asian economic crisis)N18

0.50000.54290.5206

チェチェン紛争(Chechin (Chechnia) civil war)

N19

0.41670.60810.4945

スハルト大統領(Indonesian President Suharto)

N20

0.59220.54950.5701

北朝鮮のミサイル開発放棄(Nuclear missile abandonment of North Korea)

N210.37500.55810.4486

アジアでの航空機墜落事故(Airplane crashes in Asia)

米国同時多発テロ＆（米国）経済
0.5525 (The impact of 911 terrorist attacks on America’s economy)
薬害エイズ＆ミドリ十字
0.6480(HIV-tainted blood scandal)