# On the Feature Selection and Classification Based on Information Gain for Document Sentiment Analysis

Authors: Asriyanti Indah Pratiwi; Adiwijaya
Journal: Applied Computational Intelligence and Soft Computing
 (2018)

---

## Abstract

Sentiment analysis in a movie review is the needs of today lifestyle. Unfortunately, enormous features make the sentiment of analysis slow and less sensitive. Finding the optimum feature selection and classification is still a challenge. In order to handle an enormous number of features and provide better sentiment classification, an information-based feature selection and classification are proposed. The proposed method reduces more than 90% unnecessary features while the proposed classification scheme achieves 96% accuracy of sentiment classification. From the experimental results, it can be concluded that the combination of proposed feature selection and classification achieves the best performance so far.


---

## Body

## 1. Introduction

One of the interesting challenges in text categorization is sentiment analysis, a study that analyzes the subjective information of specific object [1]. Sentiment analysis can be applied on various level: document level, sentence level, and feature level.Sentiment-based categorization in the movie review is a document-level sentiment analysis. It treats the review as a set of independent words by ignoring the sequence of words on a text. Every single unique word and phrase can be used as the document features. As a result, it constructs massive numbers of features. In addition, it also slows down the process and makes the classification task bias [2].Actually, not all features are necessary. Most of the features are irrelevant to the class label. On the other hand, a good feature for classification is the one that has maximum relevance with the output class.As feature selection in sentiment analysis is a crucial part, in this paper, we proposed an information gain based feature selection. In addition, we also proposed classification schemes based on the dictionary that is constructed by selected features.


## 2. Previous Work

There are two common approaches to sentiment analysis: machine learning methods and knowledge-based methods. Cambria [3] suggested the combination of both methods: using machine learning to provide the limitations of the sentiment knowledge. On the other hand, it cannot be applied in movie review. The sentiment knowledge such as SenticNet is highly dependent on domain and context. For example, “funny” means positive for comedy but negative for horror movie [4].Machine learning-based sentiment analysis on movie review initialized by Pang et al. [5]. Their work performed 70%–80% accuracy while the human baselines sentiment analysis only reaches 70% accuracy. In 2014, Dos Santos and Gatti [6] used deep learning method for sentence-level sentiment analysis that reached 70%–85% accuracy. Words and characters are used as sentiment features. Unfortunately, the massive constructed features resulted in a long-time computation.In order to provide robust machine learning classification, a feature selection technique is required [7]. Some researchers focus on reducing the number of features [8]. Manurung [9] proposed a feature selection scheme named feature-count (FC). FC selects n-top subfeatures with the highest frequency count. It only costs O
(
n
) to select the subfeatures. O then contrary, it may select a feature which has no relevance to the output class, since high occurrence does not indicate high relevance to the output class.Nicholls and Song [8] research and OKeefe and Koprinska [10] research proposed similar idea to select features based on the difference between document frequency (DF) in class positive and DF in class negative. It was named Document Frequency Difference (DFD). DFD selects the feature that has the highest proportion between the positive DF-negative DF difference and the total number of documents. Their research may select feature which has high difference but less relevant to the output class.Information theory-based feature selection such as information gain or mutual information was also proposed in sentiment analysis [11, 12]. In advance, Abbasi et al. proposed a heuristic search procedure to search optimum subfeature based on its information gain (IG) value named Entropy Weighted Genetic Algorithm (EWGA) [13]. EWGA search optimal subfeatures using Genetic Algorithm (GA) which its initial population is selected by information gain (IG) thresholding schemes. Compared to the other, EWGA is the most powerful feature selection so far. It selected features that achieved 88% accuracy of classification. However, it took high-cost computation.This study uses polarity v.2.0 from Cornell review datasets, a benchmark dataset for document-level sentiment analysis, that consists of 1000 positive and 1000 negative processed reviews [14]. This dataset split into tenfold cross-validation.


## 3. Information Gain on Movie Review

Information gain measures how mixed up the features are [15]. In sentiment analysis domain, information gain is used to measure the relevance of attribute A in class C. The higher the value of mutual information between classes C and attribute A, the higher the relevance between classes C and attribute A.(1)
I
C
,
A
=
H
C
-
H
C
∣
A
,where H
(
C
)
=
-
∑
c
E
C
‍
p
C
log
⁡
p
(
C
), the entropy of the class, and H
(
C
∣
A
) is the conditional entropy of class given attribute, H
(
C
∣
A
)
=
-
∑
c
E
C
‍
p
(
C
∣
A
)
log
⁡
p
(
C
∣
A
). Since Cornell movie review dataset has balanced class, the probability of class C for both positive and negative is equal to 0.5. As a result, the entropy of classes H
(
C
) is equal to 1. Then the information gain can be formulated as(2)
I
C
,
A
=
1
-
H
C
∣
A
.The minimum value ofI
(
C
,
A
) occurs if only if H
(
C
∣
A
)
=
1 which means attribute A and classes C are not related at all. On the contrary, we tend to choose attribute A that mostly appears in one class C either positive or negative. On the other words, the best features are the set of attributes that only appear in one class. It means the maximum I
(
C
∣
A
) is reached when P
(
A
) is equal to P
(
A
∣
C
1
) resulting in P
(
C
1
∣
A
) and H
(
C
1
∣
A
) being equal to 0.5. When P
(
A
)
=
P
(
A
∣
C
1
), then the value of P
(
A
∣
C
2
) results in P
(
C
2
∣
A
)
=
0 and H
(
C
1
∣
A
)
=
0. The value of I
(
C
,
A
) is varied from 0 to 0.5.


## 4. Sentiment Analysis Framework

This study uses polarity v.2.0 from Cornell review datasets, a benchmark dataset for document-level sentiment analysis, that consists of 1000 positive and 1000 negative processed reviews [14]. This dataset split into tenfold cross-validation.Figure1 shows the process of proposed sentiment analysis. The process was categorized into dictionary construction phase and classification phase. Dictionary construction phase constructs a dictionary that can be used to classify the review: positive or negative. Here are the steps of dictionary construction phase in this study: (
1
) reading the dataset, (
2
) nonalphabetic removal, (3) tokenization, (4) stopwords removal, (5) stemming (optional), (
6
) initial vocabulary construction, (7) initial feature matrix construction, (8) DF thresholding, (9) IG-DF-FS, and (10) dictionary construction.Figure 1
Classification flowchart.Similar to the dictionary construction phase, classification phase also consists of preprocessing and feature construction. On the contrary, it uses the constructed dictionary instead of selecting feature and constructs another dictionary. The result of this phase is sentiment labeled movie review.


### 4.1. IG-DF Feature Selection

Previous work on information gain [16] selects feature that has high relevance with the output class. Those features commonly appear in positive class or negative class only. Unfortunately, it may appear only a few times since the sentiment can be expressed in a various way. As a result, overfitting occurs since those features do not appear.On the other hand, DF thresholding [8, 12] selects feature that appears most in the training set. It may select feature that always appears in both classes. Those features are unnecessary since it cannot differentiate the class to which it belongs.In this study, we propose a combination of information gain and DF thresholding feature selection, named IGDFFS. IGDFFS selects a feature that has IG score equal to 0.5. It means those features highly related to one class only. These schemes succeed in reducing about 90% of unnecessary features (Algorithm1).Algorithm 1:IGDF feature selection.
(1) procedure IGDF–Feature–Selection(input: {
array of attributes A and its class C
},
output:{
positive and negative feature set
})
(2) for  each features in featureset  do
(3)calculate
I
(
C
∣
A
)
(4)end  for
(5)for  each IGscore in  I
(
C
∣
A
)  do
(6)if  I
(
C
∣
A
)
=
=
0.5  then
(7)V
o
c
a
b
u
l
a
r
y
←
V
o
c
a
b
u
l
a
r
y
+
A
(8)if  P
(
A
)
=
=
P
(
A
∣
C
p
o
s
i
t
i
v
e
)  then
(9)f
e
a
t
u
r
e
s
e
t
p
o
s
i
t
i
v
e
←
f
e
a
t
u
r
e
s
e
t
p
o
s
i
t
i
v
e
+
A
(10)else
(11)f
e
a
t
u
r
e
s
e
t
n
e
g
a
t
i
v
e
←
f
e
a
t
u
r
e
s
e
t
n
e
g
a
t
i
v
e
+
A
(12)end  if
(13)end  if
(14)end  for
(15)end  procedure


### 4.2. Classification

As it is known that entropy and information gain are commonly used in decision tree. The selected feature with the highest information gain determines the class of the review. Based on this intuition, we categorize our vocabulary into the positive feature and negative feature. A review will be classified into positive review if most of the features are positive and vice versa (Algorithm2).Algorithm 2:IG-based classification.
(1) procedure IG-based–Classifier(input: {
Sentiment Feature Vector: Vocabulary
× Number ofDocument
},  output: {
Sentiment Label: positive or negative
})
(2)  for each document in featurevector  do
(3)for  each vocabinVocabulary  do
(4)if  v
o
c
a
b
is positive – features  then
(5)p
o
s
i
t
i
v
e
←
p
o
s
i
t
i
v
e
+
1
(6)else
(7)n
e
g
a
t
i
v
e
←
n
e
g
a
t
i
v
e
+
1
(8)end if
(9)end for
(10)if  p
o
s
i
t
i
v
e
>
n
e
g
a
t
i
v
e  then
(11)c
l
a
s
s
l
a
b
e
l
←
c
l
a
s
s
l
a
b
e
l
+  ′positive′
(12)else
(13)c
l
a
s
s
l
a
b
e
l
←
c
l
a
s
s
l
a
b
e
l
+  ′negative′
(14)end if
(15)end for
(16)end procedure


## 4.1. IG-DF Feature Selection

Previous work on information gain [16] selects feature that has high relevance with the output class. Those features commonly appear in positive class or negative class only. Unfortunately, it may appear only a few times since the sentiment can be expressed in a various way. As a result, overfitting occurs since those features do not appear.On the other hand, DF thresholding [8, 12] selects feature that appears most in the training set. It may select feature that always appears in both classes. Those features are unnecessary since it cannot differentiate the class to which it belongs.In this study, we propose a combination of information gain and DF thresholding feature selection, named IGDFFS. IGDFFS selects a feature that has IG score equal to 0.5. It means those features highly related to one class only. These schemes succeed in reducing about 90% of unnecessary features (Algorithm1).Algorithm 1:IGDF feature selection.
(1) procedure IGDF–Feature–Selection(input: {
array of attributes A and its class C
},
output:{
positive and negative feature set
})
(2) for  each features in featureset  do
(3)calculate
I
(
C
∣
A
)
(4)end  for
(5)for  each IGscore in  I
(
C
∣
A
)  do
(6)if  I
(
C
∣
A
)
=
=
0.5  then
(7)V
o
c
a
b
u
l
a
r
y
←
V
o
c
a
b
u
l
a
r
y
+
A
(8)if  P
(
A
)
=
=
P
(
A
∣
C
p
o
s
i
t
i
v
e
)  then
(9)f
e
a
t
u
r
e
s
e
t
p
o
s
i
t
i
v
e
←
f
e
a
t
u
r
e
s
e
t
p
o
s
i
t
i
v
e
+
A
(10)else
(11)f
e
a
t
u
r
e
s
e
t
n
e
g
a
t
i
v
e
←
f
e
a
t
u
r
e
s
e
t
n
e
g
a
t
i
v
e
+
A
(12)end  if
(13)end  if
(14)end  for
(15)end  procedure


## 4.2. Classification

As it is known that entropy and information gain are commonly used in decision tree. The selected feature with the highest information gain determines the class of the review. Based on this intuition, we categorize our vocabulary into the positive feature and negative feature. A review will be classified into positive review if most of the features are positive and vice versa (Algorithm2).Algorithm 2:IG-based classification.
(1) procedure IG-based–Classifier(input: {
Sentiment Feature Vector: Vocabulary
× Number ofDocument
},  output: {
Sentiment Label: positive or negative
})
(2)  for each document in featurevector  do
(3)for  each vocabinVocabulary  do
(4)if  v
o
c
a
b
is positive – features  then
(5)p
o
s
i
t
i
v
e
←
p
o
s
i
t
i
v
e
+
1
(6)else
(7)n
e
g
a
t
i
v
e
←
n
e
g
a
t
i
v
e
+
1
(8)end if
(9)end for
(10)if  p
o
s
i
t
i
v
e
>
n
e
g
a
t
i
v
e  then
(11)c
l
a
s
s
l
a
b
e
l
←
c
l
a
s
s
l
a
b
e
l
+  ′positive′
(12)else
(13)c
l
a
s
s
l
a
b
e
l
←
c
l
a
s
s
l
a
b
e
l
+  ′negative′
(14)end if
(15)end for
(16)end procedure


## 5. Results and Analysis

Figure2 shows the performance previous feature selection (FFSA) [16] and proposed feature selection (IGDFFS). The results show that IGDFFS selects better features.Figure 2
Feature selection performance comparison.Proposed method selects feature that has high relevance to the output class and also has the highest occurrence. As a result, generated feature matrix has less zero value. On the contrary, the previous method may succeed in selecting high relevant features but probably takes rare features. The rare feature does not appear in another movie review document in training set and may not appear in the testing set. As a result, the generated feature matrix consists of a lot of zero value. A lot of documents which have not any features are hard to be classified.One of the feature selection objectives is to avoid overfitting. Actually, in this case, common machine learning techniques may result in overfitting. The reason is the feature matrix in testing set consists of a lot of zero values more than the feature matrix in training set. Since the features affect machine learning model, then it is hard for machine learning to fit the model to the feature matrix in the testing set.Figure3 summarizes the performance of SVM, ANN, and IG classifier. Unfortunately, SVM and ANN suffer from overfitting problems. Their testing accuracy fails in achieving 70% accuracy. Different to ANN and SVM, IGC is quite stable in any condition. IGC succeed in avoiding overfitting problems. It can be concluded that IGC as proposed classifier performs better than the current classifier.Figure 3
Sentiment classifier performance comparison.Information gain value tells how mixed a feature to the class is. IG value reaches the highest value (0.5 in this case) when the feature belongs to one class only. It means when the feature appears we make sure that the label must be positive or negative. In this case, the IG value of selected feature achieves the maximum value on average (0.5) so, it can be used for automatic classification. The specialty of proposed classification scheme is the independence from mathematical model. Since proposed classification method succeeds in avoiding overfitting, we can say that our method is better than the previous work.


## 6. Conclusion and Future Work

In order to provide better sentiment analysis system, an improvement of information gain based feature selection and classification was proposed. The proposed feature selection selects feature that has high information gain and high occurrence. As a result, it succeeded in providing feature that most probably appears in testing also. Proposed classifier used the positive and negative features obtained from the IG calculation before. Then, it takes less time than the previous classifier (SVM, ANN, etc.).The combination of information gain and document frequency in this study proposed feature selection; IGDFFS selects subfeatures that satisfy these criteria:(
1
) high relevance to the output class and (
2
) high occurrence in dataset. As a result, it constructs subfeatures that reach better performance in the classification.Compared to the current classifier, Information Gain Classifier (IGC) overcomes the recent high accuracy which belongs to EWGA (only 88.05%). It succeeded in avoiding overfitting problems in any condition. The performance of IGC is quite stable in both training and testing.We are considering to groups the words based on their relevance to positive and negative reviews. Note that there are 171,476 words that are currently used and 47,156 obsolete words in English domain (based on Oxford English Dictionary). At least a finite number of groups would be less than the total number of words.



---

*Source: 1407817-2018-02-19.xml*