Multi-aspects Review Summarization with Objective Information
Kazutaka Shimada, Ryosuke Tadano and Tsutomu Endo
Department of Artificial Intelligence, Kyushu Institute of Technology
680-4 Kawazu, Iizuka, Fukuoka 820-8502, Japan
shimada, r tadano, endo@pluto.ai.kyutech.ac.jp
Abstract
In this paper, we propose a method for multi-aspects review
summarization based on evaluative sentence extraction. We
handle three features; ratings of aspects, the   value, andthe number of mentions with a similar topic. For estimating
the number of mentions, we apply a clustering algorithm. By
using these features, we generate a more appropriate summary. In this paper, we also focus on objective informationof the target product. We integrate the summary from sentiment information in reviews and the objective information
extracted from Wikipedia. The experiment results show theeffectiveness of our method.
Keywords: sentiment analysis, multi-aspects review summarization, ob-

Senment summarizer
Mul-reviewsRangs of aspects

-idf value

The number ofmenonsesmated rangs fromthe rangs of reviews
usability

the value of words

Clustering opinions

opinion1
opinion2

word1: 1.3
word2: 0.2
word3: 0.7
the value of opinions

designSenment informaon

opinion3
opinion4

opinion1: 2.5
opinion2: 1.8
opinion3: 2.0

Integraon
Wikipedia
Fact extractor
Objecve informaon of the target

Summary

Objecve informaon

Figure 1: The outline of our method.

jective information, opinion integration.

1. Introduction
As Web services like CGMs have become widely used, people can easily post reviews for products or services. Althoughhandling these information (evaluative information) has become necessary, there exists too much information on theWeb. Therefore, extracting information that users want andsummarizing them have been expected recently. Intuitively,
we can summarize a review with traditional document summarization methods. For instance, Brandow et al. [2] havesummarized a document by extracting sentences with somefeatures such as the presence of signature words and the location in the document. For sentiment summarization, Pang
and Lee [9] have extracted all subjective sentences. They suggested that these extracted sentences could be used as summaries. However, a review basically consists of sentimentswith various aspects (i.e., “image quality” and “usability” ofa camera). Therefore, we need to extract information for each
aspect in the case of review summarization. Aspect summarization can present information without biasing to a specifictopic. We focus on multi-aspects review summarization inthis research.
Here we also focus on other information; objective information such as the market share, specifications and price ofthe target product. It is also important for the summarization.

Integrating a sentiment summary from reviews with objective
information leads to improvement in the quality of the final
summary.
In this paper, we propose a method for generating a summary that contains sentiment information and objective information of a product. Figure 1 shows the outline of our
method. For the sentiment summarization task, the method
is based on an extraction approach. Therefore, we need to
discuss which sentences are important and how to extract important sentences. In the case of treating multi-reviews, weneed to handle the redundant information. Pang and Lee [10]have reported that while in traditional summarization redundant information is discarded, in sentiment summarization redundancy indicates the importance of opinions. Therefore,
we treat redundancy as a feature for decision of importantsentences. We assume that reviews we treat in this research
have multiple aspects and a reviewer gives ratings of the aspects (i.e., 0 to 5 stars). Reviewers also write free comments
about the target. We leverage three features: ratings of aspects of reviews, the   value, and the number of mentions with a similar topic. We apply a clustering algorithm tosentences to measure the number of mentions with a similar
topic. Then, we generate a more appropriate summary by using those features. For the objective information extraction,
we treat a Wikipedia entry about the current target product.
We extract relative information of the product by using key-

words in the sentiment summary and structural information,such as itemization, in Wikipedia.

2. Features for sentence extraction
For generating a summary, we define the following three features: (1) ratings of aspects, (2) the   value, and (3) the
number of mentions with a similar topic. The following sections describe how we treat these features.

2.1. Ratings of aspects
If we generate a summary, the summary needs to contain theproper balance of whole opinions in the reviews. For instance, if we summarize only positive opinions, the summary
cannot tell readers negative opinions of a target.
We focus on ratings of aspects given to reviews as a feature
to deal with this problem. We assume that a reviewer writes
comments corresponding to the ratings. For instance, if a reviewer gives 1 star for one aspect, the reviewer writes negative comments for the aspect. We assign ratings of aspects ofa review to evaluative sentences included in the review. Eachevaluative sentence has the rating corresponding to the aspectof the sentence. We use the pair of the evaluative sentence
and the rating to consider the distribution of the ratings forsummarization.

2.2. The   value
The   algorithm is used as a major algorithm for manytasks such as important sentence extraction. This algorithm
features words which only appear in a target document as
more important. We similarly apply the   algorithm tocompute the importance of an evaluative sentence. First, we
divide sentences in reviews to morphemes by using a morphological analyzer 1 . Then, we define the   value of aword  in target reviews  as below. Note that we treat only
content words (except words such as suffixes and pronouns).
¾    
¾    
¾   ´ µ

 







(1)

(2)

(3)

where    is the frequency of  for  ,   is thenumber of words belonging to  ,   is the number of all
reviews, and  ´ µ is the number of reviews including.
1 We used Mecab. http://mecab.sourceforge.net/

Next, we compute the importance of an evaluative sentence
by using the   value of a word. We denote an evaluative
sentence by   ½  ¾  , where each is a word whichhas   value in the sentence. The importance of an evaluative sentence   is as below:
 

¾

 

 

(4)

where   is the total number of in the sentence. This is theimportance of an sentence based on feature words.

2.3. The number of mentions with a similar topic
We treat multi-reviews for the target of summarization. In
this case, some reviewers might write similar opinions. Thesesimilar opinions have possibilities to be redundantly extracted
as a summary. On the other hand, the opinion mentioned bymany reviewers is important. We need to handle redundant
information regarding as a feature to determine an importantsentence. Therefore, we aim to integrate similar opinions byclustering them.
In this paper, we apply the  - algorithm which is
widely used as the clustering algorithm because of its simplicity. Since the  - algorithm is a non-hierarchicalmethod, firstly we need to specify how many clusters we divide. However, It is difficult to know the optimum numberof divided clusters beforehand. Seki et al. [11] have estimated the valid number of clusters by statistically evaluating
the clustering result. We apply their algorithm to our task.
We divide evaluative sentences to morphemes and construct a vector space using these morphemes as features. Notethat we treat content words, adjectives, and verbs. If the feature has the   value, a score of each feature is the  
value computed in Section 2.2. If a feature does not havethe   value, we simply assign the frequency within theevaluative sentence as the score. Besides, we introduce theconcept of centrality of the word which has been reported by
Ishii et al. [4] to characterize features. This method assumes
that words such as a subject case and an objective case in a
sentence indicate a topic of the sentence. We apply the concept to our task and weight central words in the evaluative
sentence.
By clustering evaluative sentences based on these algorithms, we can generate some clusters including similar sentences. However, we found that clusters generated by ouralgorithm tended to be divided too much in a preliminary experiment. It denotes that similar opinions which should belong to the same cluster belong to other clusters. Therefore
we revise the clusters by using co-occurrence of representative words of each cluster [13].

3. Sentiment Summarizer

Wikipedia

In this section, we describe how to generate a summary basedon three features mentioned in Section 2.. We compute theimportance of each cluster by integrating the   value ofeach sentence and the number of mentions. The importanceof a cluster    is as below:     Ë     

Structural information: table

Structural information: itemization(5)

where    Ë   is the average of   belonging toa cluster  .   is the total number of sentences in the cluster.
   is the importance which means both the importanceof feature words and the number of mentions.
Besides, we treat ratings of aspects to reflect the properbalance of whole opinions of reviews. The process for thesentence extraction is as follows:
1. identify a representative sentence which has the top  from each cluster,
2. classify representative sentences into the rating to which
the sentence belongs,
3. extract sentences with high    on the basis of the
distribution of the total number of representative sentences belonging to each rating.
As the representative sentence, we select a sentence which
is close to the centroid of the cluster. However, each cluster often contains sentences with different polarities; positive
and negative. Therefore, we judge the major polarity of eachcluster first. Then, we select the sentence which is close to
the centroid and contains the polarity.

KeyWik: Grand Prix mode
Sentece: Players compete against one another in a themed cup.
KeyWik: Time Trial / Time Attack
Sentence: The goal is to achieve the fastest time on the selected track.
KeyWik: VS. mode
Sentence: Multiple players can compete against each other in a single race.

Figure 2: The KeyWik extraction process.

In English: The worst point is that it does not include benefits after beating the game.

Representative sentences

Cluster display button

Click: Keyword [beat the game]

Click

Keyword and its supplement information

Sentences in the cluster

In English: The purpose of the game is to rescue Princess Peach from Bowser.

In English: It isn't offered benefit after finishing the game.

4. Integration
In this section, we explain an integration approach of a sentiment summary and objective information which is related
to it. In this paper we regard specifications and explanations
about the target product as objective information. Integrating
a sentiment summary from reviews with objective information leads to improvement in the quality of the summary.
In this paper, we handle a Wikipedia entry about the target
product. The integration process consists of two processes;
detection and alignment of KeySum (keywords in a sentimentsummary) and KeyWik (keywords in a Wikipedia entry) .
KeySum is nouns with the high   values in a sentimentsummary. The number of KeySums in each sentence is limited byof nouns in a sentence
Our system extracts sentences including the KeySum fromthe Wikipedia entry. On the other hand, KeyWik is detected by using structural information, such as itemization, in

Figure 3: Summary output interface.
Wikipedia. Figure 2 shows an example of the process. If thesentiment summary contains the KeyWik, our system aligns
words in the sentiment summary and the KeyWiks.
Figure 3 shows the output of our system. The system candisplay (1) representative sentences in each aspect, (2) supplement information and (3) sentences belonging to each representative sentence. Each representative sentence contains
the importance score (  ) and the rating. By clicking a
keyword in the summary, our system displays sentences extracted from Wikipedia as supplement information. The supplement sentences are displayed in descending order of theimportance. It is based on the concept of centrality of theword and is the same as the approach, which was proposedby Ishii et al. [4], in Section 2.3. If a user pushes a “cluster
display” button, he/she can access to sentences which belong

Table 1: A summary by the proposed method for Addiction (a).
Representative sentence
I’m bored with in a few days because it is stressful.
The worst point is that it does not include benefits after beating the gameThe number of hidden items is good.
Hidden stages after completing the game are scarce.
I enjoyed collecting Star coins.
Big mushroom item is very funny idea.

to the cluster.

5. Experiment
In this section, we evaluated our summarization. First, we describe the data set for the experiment. Then, we compare oursummary with a manual summary as quantitative evaluation.
Finally, we evaluate our system with objective information
qualitatively.

5.1. Data set
We used game review documents which Shimada et al. [12]used for evaluative documents classification. The review documents were extracted manually from the Web site 2 . Seven
evaluative criteria are given to each review, i.e., Originality(o), Graphics (g), Music (m), Addiction (a),Satisfaction (s), Comfort (c), and Difficulty (d).
The review documents include reviews for 49 games. The
numbers of the all reviews are 4,174 reviews. We chosen
three of the Nintendo DS software 3 as the target data. They
consisted of 170, 130, 24 reviews, respectively. We randomly selected approximately 450 sentences from each software. Then, three annotators ( ½ , ¾ , ¿ ) annotated the sentences. For quantitative evaluation, annotators manually generated summaries. Each annotator extracted 50 sentences asa summary from them for each software.

 

 

2.46
4.26
3.45
4.96
8.43
3.03

0
1
2
3
4
5



0.00
1.80
2.00
3.5
3.36
5.00

polarity-adjusted sentence extraction (See Section 3.). However, one of the problems in the result of Table 1 was that
there existed the sentence to which its rating did not correspond. For example, although “The number of hidden items
is good.” was usually considered as a positive opinion, the rating of the review containing the sentence was 2. The reason
was that most of the sentences in the review containing therepresentative sentence had negative opinions for this aspect.
This is due to the fact that opinions of reviewers sometimes
might be inconsistent with the ratings. We need to handle thenon-consistence of ratings if we treat the rating information.
Next, we compared our summaries with manual summaries. We used ROUGE-N [5] for evaluation of summaries.
It indicates an n-gram recall between reference summaries
and a candidate summary. ROUGE-N is computed as follows:
ROUGE-N¾  ¾ ¾ À Æ ¾
À

Æ

 
  

 

(6)



where  is a set of reference summaries,   is the  lengthn-gram,    is the frequency of the   in the reference
summary, and     is the maximum number of n-grams
co-occurring in a candidate summary and a set of reference
summaries.
Table 2 shows ROUGE-1 between our summaries andmanual summaries. The baseline method was based on onlythe   value, that is a method without clustering. Thosescores are the average scores in three annotators. Since the
distribution of each aspect was different, we computed theweighted average   . We weighted the number of sen5.2. Evaluation of a summary
tences in reference summary for each aspect. AnnoR denotes
First, we discuss output summaries from our method. Table 1 the weighted average ROUGE score between annotators. We
shows an example of a summary for the aspect Addiction regarded  ½ as the reference summary and the scores are the(a). It contains some representative sentences and their average of  ¾ and ¿ .
Even the scores between annotators were not high. This  .  is the actual rating of the review containing the representative sentence, “ ” is the average of the result shows that it is difficult to generate a same sentimentsummary. For MC and FS, our method outperformed theratings in the cluster containing the representative sentence.
The value of the   was close to that of   . This re- baseline in most aspects and   . On the other hand, thesult shows the effectiveness of the sentiment summarizer with ROUGE score of our method was slightly low as comparedwith that of the baseline for SM.
2 http://ndsmk2.net/
3 In this paper, there were “SM (New Super Mario Brothers)”, ”MCHowever, the evaluation with ROUGE has a problem. The(Mario Cart)”, “FS (Family Stadium)”.
ROUGE is based on correspondence between the output andTable 2: ROUGE-N between our summaries and manual summaries.
Name
SM
MC
FS

MethodBaseline
Proposed
Baseline
Proposed
Baseline
Proposed

a
0.301
0.275
0.304
0.492
0.384
0.430

c
0.506
0.453
0.310
0.439
0.416
0.454

d
0.357
0.430
0.468
0.393
0.211
0.404

m
0.476
0.303
0.289
0.341
0.644
0.644

g
0.095
0.205
0.363
0.347
0.338
0.101

o
0.373
0.341
0.388
0.456
0.371
0.411

s
0.335
0.350
0.408
0.498
0.451
0.441

 

0.349
0.337
0.361
0.424
0.402
0.412

AnnoR
0.460
0.371
0.522

Table 3: Semantic agreement between our summaries and manual summaries (%).
Name
SM
MC
FS

MethodBaseline
Proposed
Baseline
Proposed
Baseline
Proposed

a
14.3
28.6
25.0
37.5
25.0
50.0

c
37.5
31.3
33.3
44.4
13.6
50.0

d
21.4
50.0
40.0
0.0
14.3
42.9

manual summaries. Therefore, the value becomes low in thecase that surface expressions are different even if the senseof sentences is similar. In this experiment, we evaluated themethods with semantic agreement between our outputs andmanual summaries. First we displayed two sentences, namely
our output and a manual output, to a test subject. Then, thetest subject judged whether the outputs were similar in termsof content. The result shows Table 3. The number of test subjects was 2 persons. As compared with the result of Table 2,
i.e., ROUGE, the proposed method generated higher scores.
This result shows the effectiveness of the proposed method ascompared with the baseline.

5.3. Evaluation of our system
We evaluated the effectiveness of object information and oursystem. This experiment was qualitative evaluation and thenumber of test subjects was four persons. The test subject
scored 1 (bad) - 5 (good) points for each questionnaire entry.
The result shows Table 4. The average score was 4.3.
In particular, the score of “effectiveness for content understanding” as a summary was high. This result shows the effectiveness of integration between object information and asentiment summary. By using our system, a user can easily
understand the product in terms of sentiment and objective
information.
On the other hand, there were some negative commentsfrom the test subjects. The 1st comment was a problem of
the clustering process. The clustering process did not deal
with the polarity of each sentence. Sentences with a differ-

m
25.0
25.0
0.0
50.0
33.3
33.3

g
33.3
66.7
20.0
20.0
0.0
16.7

o
27.3
36.4
31.8
45.5
40.0
30.0

s
20.0
25.0
22.2
33.3
37.5
29.2

 

25.5
37.6
24.6
33.0
23.4
36.0

Table 4: Evaluation of our system
Effectiveness of objective information
Correspondence between supplement
information and content
Adequateness of selection of KeySum and KeyWik
Effectiveness for content understanding
Necessity of objective information
Effectiveness of our system
As informative summary
Easy-to-understandAverage

4.0
4.2
4.6
4.4
4.2
4.6
4.3

ent polarity were occasionally contained in a cluster; e.g., thepolarity of a representative sentence in a cluster was positive and the cluster contained some negative sentences. To
improve the clustering process is one future work. Anothernegative comment is concerning the keyword extraction fromsentiment reviews for the integration process. There was lackof keywords for the supplement. To generate more appropriate integrated summaries, we need to consider the extraction
process of target words for the supplement.

6. Related work
Meng and Wang [7] have extracted aspects from the specification of the target product and summarized reviews with hier-archic structures. As a result, they could extract appropriate
aspects for the products. However, the generated summary
did not include detailed opinions about the product. In contrast, our method can treat detailed information by extracting
important sentences with feature words.
Blair-Goldensohn et al. [1] have computed a polarity valueof sentences based on the maximum entropy method with
WordNet and rating information. They extracted evaluative
sentences with a high polarity value preferentially and generated a summary. As the advantage of their method, it could
estimate the polarity of sentences with high accuracy. However, it is not always true that sentences with high polarity
values are appropriate for a summary. They also did not treat
redundancy of the summary.
Takamura and Okumura [14] have proposed a document
summarization method based on the budgeted median problem. Nishikawa et al. [8] have proposed a opinion summarization method handling content and coherence simultaneously. These methods were effective but did not deal with
objective information.
Lu and Zhai [6] have introduced the concept of aspects toa PLSA model. They integrated expert reviews and ordinary
opinions scattering in the Web. Opinions which should be integrated are identified by measuring the number of mentionsin the Web. Although their method are very effective, their
purpose is to add more information, such as similar and supplementary opinions, to the base review.
In this paper, we did not discuss the identification of the
aspect of sentences. However, it is important to identify
the aspect information for the sentiment summarization task.
Hadano et al. [3] have identified an aspect of an evaluative
sentence with machine-learning approaches. We need to introduce such a method to our task in the future.

7. Conclusion
In this paper, we focused on the multi-aspects review summarization. We handled three features; ratings of aspects,the   value, and the number of mentions with a similar
topic. We used a clustering method to integrate similar opinions. The experimental result showed that we could integrate
similar opinions and it led to the redundancy elimination of asummary.
In addition, we handled objective information for the summarization task. Supplying objective information led to improve the content understanding of a summary. The resultin the qualitative evaluation showed the effectiveness of oursystem that integrated a sentiment summary with objective
information.
Future work includes (1) more appropriate decision of representative sentences, (2) handling not only Wikipedia but
also other information sources and (3) computing a confidence as objective information.

Acknowledgment
This work was supported by Kayamori Foundation of Informational Science Advancement.

References[1] Sasha Blair-Goldensohn, Kerry Hannan, Ryan McDonald, Tyler Neylon, George A. Reis, and Jeff Reynar. Building a sentiment summarizer
for local service reviews. In Proceedings of WWW 2008: NLPIX Workshop, 2008.
[2] Ronald Brandow, Karl Mitze, and Lisa F. Rau. Automatic condensationof electronic publications by sentence selection. Information Processing and Management, 31(5):675–685, 1995.
[3] Masashi Hadano, Kazutaka Shimada, and Tsutomu Endo. Aspect identification of sentiment sentences using a clustering algorithm. In Proceedings of PACLING 2011, 2011.
[4] Hiroshi Ishii, Rihua Lin, and Teiji Furugori. An automatic text summarization system based on the centrality of word roles in sentences(in japanese). Information Processing Society of Japan SIG Notes,
2001(20):83–90, 2001.
[5] Chin-Yew Lin. Looking for a few good metrics: Automatic summarization evaluation – how many samples are enough? In Proceedingsof NTCIR Workshop 4, 2004.
[6] Yue Lu and Chengxiang Zhai. Opinion integration through semisupervised topic. In Proceedings of WWW 2008, pages 121–130, 2008.
[7] Xinfan Meng and Houfeng Wang. Mining user reviews: from specification to summarization. In Proceedings of ACL-IJCNLP 2009, pages
177–180, 2009.
[8] Hitoshi Nishikawa, Takaaki Hasegawa, Yoshihiro Matsuo, andGenichiro Kiku. Opinion summarization with integer linear programming formulation for sentence extraction and ordering. In Proceedings of the 23rd International Conference on Computational Linguistic(COLING ’10), 2010.
[9] Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis
using subjectivity summarization based on minimum cuts. In Proceedings of Association for Computational Linguistics (ACL), pages 271–
278, 2004.
[10] Bo Pang and Lillian Lee. Opinion mining and sentiment analysis.
Foundations and Trends in Information Retrieval, 2 (1-2):1–135, 2008.
[11] Tsunehito Seki, Kazutaka Shimada, and Tsutomu Endo. Acquisition ofsynonyms using relations in product. Systems and Computers in Japan,
38(12):25–36, 2007.
[12] Kazutaka Shimada and Tsutomu Endo. Seeing several stars: a rating
inference task for a document containing several evaluation criteria. In
Proceedings of PAKDD 2008, 2008.
[13] Ryosuke Tadano, Kazutaka Shimada, and Tsutomu Endo. Multiaspects review summarization based on identification of important
opinions and their similarity. In Proceedings of the 24nd Pacific Asia
Conference on Language, Information and Computation (PACLIC24),
pages 685–692, 2010.
[14] Hiroya Takamura and Manabu Okumura. Text summarization model
based on the budgeted median problem. In Proceedings of the 18th
ACM Conference on Information and Knowledge Management (CIKM2009), pages 1589–1592, 2009.