言語処理学会 第25回年次大会 発表論文集 (2019年3月)

Fake Review Detection Focusing on Emotional Expressions andExtreme Rating
Tavanleuang VANTA Masaki Aono
Toyohashi University of Technology
Department of Computer Science and Engineering
vanta@kde.cs.tut.ac.jp aono@tut.ac.jp
review

1. Introduction

detectionusing

machinelearning

algorithms

toincluding SVM, NB, RF, and Decision Tree. The authors

historical reviews of a product/service to help them makeuse down-sampling to extract a balanced data set from thedecision. Booking a hotel room, deciding to go to a new

original Yelp data set collected by Mukherjee et al. andrestaurant, or picking a movie to watch, makes online

extract 21 verbal and 26 non-verbal behavioral features inreviews useful. However, the usefulness of reviews is

both hotel and restaurant reviews. Linguistic features such

hampered by fake reviews which are often posted byas n-grams, POS, and other non-behavioral features weresponsored reviewers. Fake reviews can be hard to spot by

not used in their study.

Inexperienced

buyers/customers

tendtoreferhuman eyes and the quantity and quality of them can impact

Ren and Zhang [6] proposed a neural network composedbusiness, thus, research of fake review detection has beenof Convolutional Neural Network (CNN) connected to a bi-

conducted in recent years using machine learning approach.

directional Gated Recurrent Neural Network (GRNN). Ren

In this paper, we extract features from review data using

and Ji’s [7] subsequent work indicated that the best model

only review texts and their ratings. We create some new

had the review representation with the POS, n-grams, anduseful features while adopt some useful literature ones, andpsychological features (LIWC)
Li et al. [8] created a deep learning model named

employ them as inputs in our proposed system for fake

Sentence Weighted Neural Network (SWNN) to detect fakereview detection.

reviews in hotel, restaurant, and doctor domains. On a

2. Related Workmixed-domain

Jindal and Liu [1] crawled non-labeled review datafrom Amazon.com and manually labeled them based on

setting(allthreedomains)thebestperformance was achieved by SVM with unigram, POS, and
LIWC features.

linguistic, behavioral, and relationship among reviews,

It is important to note that most existing researches

reviewers, and products. Ott et al. [2] collected data

exploit features from all types of data including not only

composed of true reviews from TripAdvisor (20 mostreview texts, but also so-called “metadata” such as user

popular hotels) and fake (positive) reviews of the samedata, product data, and business data.

hotels from Amazon Mechanical Turk (AMT). The authors
definedthefeatures

suchaspart-of-speech(POS),

3. Dataset
The dataset used in this paper is restaurant reviews thatunigrams, bigrams, trigrams, and psychological cues, andclassifiers,

were collected by A. Mukherjee, et al. [3] from Yelp. This

demonstrating that the classifiers outperformed humanrestaurant reviews dataset consists of two sets, YelpNYC

judges.

and YelpZip. The smaller set (YelpNYC) covers only

used

thefeaturesforSVM

andNB

Mukherjee et al. [3] later found substantial differences

reviews of the restaurants located in NYC, while the larger

between fake reviews submitted to a review website andset (YelpZip) covers reviews collected for the restaurants

those artificially generated by AMT. The author used SVM

based on Zipcodes of continuous region of the US map,

trained on AMT to classify Yelp data, a commercial

including NJ, VT, CT, and PA. YelpNYC contains 359,052

platform that filters suspicious reviews, and yieldedreviews, while only 10.27% of them are fake reviews. We

maximum accuracy of 54%. Mukherjee et al. and Li et al.

created a balanced review data in terms of the number of[4]

fake/true reviews from YelpNYC and use them in ouraffirm

thatAMTgenerated

datacannot

beexperiment.

representative of all types of real-life fake reviews.
Zhang et al. [5] studied both verbal and nonverbal

Yelp has a filtering algorithm in place that identifies

behaviors to identify which contributed to the most to fake

fake/suspicious reviews and separates them into a filtered― 422 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

list. The filtered reviews are also made public; the Yelp

divided by number of words in each review. We set thepage of a business shows the recommended reviews, while

calculation

it is also possible to view the filtered/unrecommendedcount)/(word count +1) to avoid the division by zsero sincereviews through a link at the bottom of the page [9].

formulafor‘word

density’as(charsome reviews might not have punctuations (e.g. “This place

We shuffled the 73,770 reviews to avoid poorly-orderedis great”), and will have the number of sentence as “0”.

fake/true reviews before splitting them to training set(51,639 reviews) and test set (22,131 reviews)

A python library TextBlob, which allows us to perform
natural language processing (NLP) for part-of-speech task,
is used to extract linguistic characteristics of fake and true

4. Proposed Method
Our proposed method to detect fake reviews from text
data can be divided into feature selection part andmethodology part.

reviews such as ‘noun count’, ‘verb count’, ‘adjectivecount’, ‘adverb count’, ‘pronoun count’. Ratio of thesevalues are themselves divided by ‘word count’.
‘ratio of positive words’ and ‘ratio of negative words’ iscalculated by the number of positive/negative words

4.1. Feature Selection
All of the custom features extracted from review texts

divided by number of words in each review, positive words
and negative words are considered based on Opinion

are listed in Table 1.

Lexicon [10].

Table 1 List of features1
rating2char count3
word count4
numeral token count5
punctuation count6
sentence count7
title word count8
uppercase word count9
word density
10 average sentence length
11 noun count12 verb count
13 adjective count
14 adverb count15 pronoun count
16 ratio of nouns
17 ratio of verbs
18 ratio of adjectives
19 ratio of adverbs
20 ratio of pronouns
21 ratio of uppercase words
22 ratio of numeral words
23 ratio of positive words
24 ratio of negative words
25 extremity of rating

Reviews which have rating of 1, 4, or 5 are considered
as extreme rating and will be assigned a value of ‘1’, Forreviews which have rating of 2 and 3, ‘0’ will be assigned.

4.2. Methodology
To accomplish our goal, we constructed the system asshown in Figure 1, where P1, P2, and P3 refer to theprediction results (‘true’ or ‘fake’) from SVM, MLP, andCNN+LSTM respectively. P4 refers to the prediction
obtained by using majority voting method over prediction
results of SVM, MLP, and CNN+LSTM.
Train Data
Review text + rating

Preprocessing
Features
Custom features+BoW features
Review text

Note that ‘rating’ represents the rating of the restaurant
given by the reviewer and ‘word density’ is the average

fastText
Embedding sClassifiers

SVMMLPCNN+LSTM

Predictions

P1

P2

P3

length of the words used in each review.
We obtain ‘rating’ of each review directly from thedataset. For the second to eighth features in Table 2, thetotal numberof characters,

words,

numeraltokens,punctuation, sentence, title words, uppercase words, inVoting

each review are counted respectively for ‘char count’,P4‘word count’, ‘numeral token count’, ‘punctuation count’,‘sentence count’, ‘title word count’, ‘uppercase word

Figure 1 − Proposed system for fake review detection

count’.
‘word density’ is calculated by number of characters― 423 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

4.2.1. Feature Extraction and Data Preprocessing

4.2.4. Majority Voting

We employ the method mentioned in 3.1 to extract

Here we use the simplest case of majority voting whichuseful features from the review text and build featureis hard voting. We predict the class label

vector for each review. Feature scaling is applied tovoting of each classifier,standardize the range of features of data.
Customer reviews usually contain unimportant words,for instance, stem words (‘the’, ‘to’, ‘on’, etc) and time,

Assuming each of our classifier give us the prediction
results of a review as follows:classifier 1 → Fake (0)indicators (’00:30’, ‘12’, etc) which occur frequently

classifier 2 → True (1)across reviews. We perform the preprocessing tasksclassifier 3 → True (1)including lowering capital letters, removing punctuations,0, 1, 1

removing stop-words, and lemmatizing words to clean thereview texts.

via majority:

1

Via majority vote, we can obtain another prediction
result which is ‘1’, this fourth prediction result can be seenas the output of ensemble vote classifier, provided that

4.2.2. Bag-of-Words Features
Each unique word in the preprocessed review text isthree of the classifiers give high and high accuracy. We willconsidered as a feature. After a dictionary of all words areneed to evaluate this method to see whether it will improvecreated, a word-review matrix is constructed. Words thator worsen the overall result.

appear too infrequently in the reviews are likely to be
misspells which are not useful and can introduce noise tothe models. Thus, we ignore terms that appear in less than
three reviews. Furthermore, words like “and” or “the”
appear frequently in all reviews, therefore we apply Term
Frequency-Inverse

Document

Frequency

5. Evaluation
In this section, we describe the evaluation criteria,
classification results, and the feature importance.

5.1. Evaluation Criteria(TF-IDF)

Analogous to Ott et al. [2], we choose Accuracy,

weighing technique to the matrix. After testing withPrecision, Recall, and F1 Score as evaluation criteria,different numbers of features, we found that using 20,000

which is defined in (1), (2), (3), and (4) respectively.

features give the most effective for our dataset.
1

4.2.3. Classification Models
In our experiment, Support Vector Machine (SVM),

2

Multi-Layer Perceptron (MLP) model, and Long ShortTerm Memory (LSTM) model (with a 1D Convolutional

3

Neural Network (CNN) layer) are used.
We choose linear SVM because it has been proved to be1

a powerful classifier and is suited for 2-class problem. Weconcatenate the feature matrix of 4.2.1 with BoW features
as described here before feeding them to SVM as input.
The second model is the MLP neural network. This
network consists of two or more fully-connected neuralnetworks, which are supposed to take care of thecorrelation between all the inputs features during thetraining stage.
The third model embedded in our system is recurrent
neural network LSTM which is known to perform well on
text

data.

Themodelaccepts

the“fastText”

24The larger these values, the better the classifier is. TP(True Positive) refers to the number of positive tuples
classified correctly as positive by the classifier; TN (TrueNegative) refers to the number of negative tuples classified
correctly as negative by the classifier; FP (False Positive)refers to the number of negative tuples wrongly labeled as
positive; and FN (False Negative) refers to the number ofpositive tuples wrongly labeled as negative.

5.2. Classification Resultword

The proposed custom (handcrafted) features used asembeddings, whose outputs are fed into the subsequent 1D

inputs to all the classifiers as well as BoW features (20,000convolutional neural network (CNN) which should takeunigrams) are shown in Table 2. Here, we set the “baseline”

care of the locally frequent patterns, followed by the LSTM.

method as an approach using only BoW features with SVM
classifier. The classification using our proposed custom― 424 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

features of SVM, MLP, and CNN+LSTM are Proposed 1,A real-life dataset of 73,770 reviews and their ratings wasProposed 2, and Proposed 3 respectively. The Majority

used to train and test the models. Accuracy, Precision,

Voting result of the three models is Proposed 4.

Recall, and F1 Score of evaluation measures were used toTable 2.

verify the experimental results. The classification results

Classification Results Across All Methods

showed that useful custom features extracted from only

MethodAccuracyPrecision

RecallF1Baseline

0.747

0.751

0.747

0.747

Proposed 1

0.773

0.775

0.773

0.773

Proposed 2

0.775

0.776

0.775

0.774

Proposed 3

0.784

0.784

0.784

0.784

other business to see how the performance of the system

Proposed 40.7830.784

0.783

0.783

will change. Furthermore, we would like to add metadata

review texts can boost the models’ performance compared
to using BoW features alone. In the future, we hope toexperiment with a larger dataset as well as the dataset ofsuch as reviewer-centric features obtained from user data−
The overall results including accuracy, precision, recall,maximum reviews in a day, percentage of positive/negativeand F1 score improved by 2.2-2.6% with our proposed

reviews, and similarity of the user ’s reviews, IP address;

features compared to the baseline approach. Moreover, theand product/service data which have been used in related

score of MLP and CNN+LSTM models indicate a betterto boost the performance and accuracy.

performance where CNN+LSTM yields the best result. On

Acknowledgment

the other hand, the predictions from voting of the three

A part of this research was carried out with the supportclassifiers give a slightly lower score than of theof the Grant-in-Aid for Scientific Research (B) (issue

CNN+LSTM model.

number 17H01746).

5.3. Feature Importance
The effectiveness of 4.2.1 and 4.2.3 features on fakereview detection performance can be simulated by Random
Forest model, which is renowned for easily estimating theimportance among features. The top 20 most effective
features and their relative importance values are shown in
Figure 4. Note that BoW features are written within single
quotes.

Figure 4 – Top 20 effective features

6. Conclusion
In this paper we investigated the effects of applying
many useful features which were extracted from review
texts and ratings to be used in SVM, MLP, and CNN+LSTM.

References[1] Jindal, N. & Liu, B. (2008), Opinion spam andanalysis, in ‘Proceedings of the 2008 International
Conference on Web Search and Data Mining’.
[2] Ott, M., Choi, Y., Cardie, C. & Hancock, J. T. (2011),‘Finding deceptive opinion spam by any stretch of theimagination’.
[3] Mukherjee, A., Venkataraman, V., Liu, B. & Glance,
N. (2013), ‘What yelp fake review filter might bedoing?’.
[4] Li, J., Ott, M., Cardie, C. & Hovy, E. (2014), Towards
a general rule for identifying deceptive opinion spam,in ‘Proceedings of the 52nd Annual Meeting of theAssociation for Computational Linguistics (Volume 1:Long Papers)’, Association for Computational
Linguistics, Baltimore, Maryland, pp. 1566–1576.
[5] Zhang, D., Zhou, L., Kehoe, J. L. & Kilic, I. Y. (2016),‘What online reviewer behaviors really matter?
effects of verbal and nonverbal behaviors on detectionof fake online reviews’.
[6] Ren, Y. & Zhang, Y. (2016), Deceptive opinion spam
detection using neural network, in ‘Proceedings ofCOLING 2016, the 26th International Conference onComputational Linguistics: Technical Papers’.
[7] Ren, Y. & Ji, D. (2017), ‘Neural networks fordeceptive opinion spam detection’.
[8] Li, L., Qin, B., Ren, W. & Liu, T. (2017), ‘Document
representation and feature combination for deceptive
spam review detection’, Neurocomputing 254, 33–41.
[9] Shebuti Rayana and Leman Akoglu (2016),’
Collective Opinion Spam Detection using Active
Inference’.
[10] Bing Liu, Minqing Hu and Junsheng Cheng. ‘Opinion
Observer: Analyzing and Comparing Opinions on theWeb.’, Proceedings of the 14th international World
Wide Web conference (WWW-2005), May 10-14,2005, Chiba, Japan.

― 425 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.