What Makes a Review Encouraging: Feature Analysis ofUser Access Logs in a Large-scale Online Movie Review Site
Kakeru Ito∗

Yoshiyuki Shoji

Tokyo Institute of Technology
Tokyo, Japan
kakeru@sw.it.aoyama.ac.jp

Aoyama Gakuin University
Kanagawa, Japanshoji@it.aoyama.ac.jp

Sumio Fujita

Martin J. Dürst

Yahoo JAPAN Corporation
Tokyo, Japansufujita@yahoo-corp.jp

Aoyama Gakuin University
Kanagawa, Japanduerst@it.aoyama.ac.jp

ABSTRACTThis paper reveals the characteristics of the reviews that encourage readers to watch the reviewed movie by analyzing large-scale
access log data. We assume that some of the reviews that users saw
just before they clicked the links to a streaming site contain factorsthat help users decide whether they watch that movie. Our method
used a random forest classifier trained to determine whether a review encouraged a movie-watching behavior. We conducted featureimportance-based analysis using three types of features: review
itself, item, and reviewer. We analyzed 70,000 user behaviors fromYahoo! Movies (a movie review site in Japan) and Gyao! (a movie
streaming site in Japan). Through a cross-validation experiment,
the classifier was able to classify encouraging reviews with an Fscore of 0.78, and mainly the features about the item contributed tothe classification performance. An additional subjects experimentconfirmed that these features contribute to the review’s usefulness.

KEYWORDS
Online Review, Access Log Analysis, Random Forest, Feature Study1

INTRODUCTION

The total number of movies released by 2021 exceededs 570,000 1 .
Assuming each movie is roughly two hours long, it would take 130
years to see all movies. No one can watch all these movies in his
or her lifetime. People need to continuously make quick decisions
about which movies to watch and which ones not to watch while
they are alive.
Online review sites are an essential information resource to help
us make such decisions in our daily life. We regularly read reviews
about products on shopping sites, about videos on streaming sites,
about facilities on maps, about schools, doctors, individuals, and soon. On the other hand, not all of the reviews posted on these review
sites are useful for decision-making. Various people write reviews;
some are well-written, some are useless. It is difficult to pick out only
the information that can be used efficiently for decision-making
from review sites.
Nowadays, there is a large number of reviews for each product.
It is clear that reading many reviews requires much time to makea decision. We consider estimating the helpfulness for decisionmaking of a review by analyzing how the review encouraged users’
page transition. Many studies discussed the usefulness of reviews
and what characteristics make a review useful. However, in general, these studies derive the degree of usefulness of a review fromwhether readers clicked the “helpful” button [6, 18, 22]. The usefulness of these reader polls does not necessarily equate to theirability to be used in actual decision-making. Consider when readers
will press the “helpful” or “unhelpful” button. For example, some
paid trolls may press the “helpful” button on a highly rated review
of a product they support as an act of stealth marketing. This kindof voting behavior is not necessarily valuable for decision-making
from the average user’s perspective. Therefore, we suggest thatusers not click the “helpful” button when they read a really usefulreview. In this study, we assume that users who read really usefulreviews would purchase the product before clicking the “helpful”
button.
In this study, we analyzed actual large-scale log data in orderto estimate the degree that reviews are useful for decision-making.
First, we collected access logs of people actually watching movies
on the streaming site immediately after reading some reviews. The
actual access log was taken from Yahoo! Movies2 , one of the biggest
online movie communities in Japan, and Gyao3 , a well-known
movie streaming service in Japan, were used for the analysis (seeFigure 1). Next, we learned a random forest classifier that separates
the reviews that encouraged movie-watching behavior from otherreviews. Three different types of features represented a review;• Features related to the review itself are characteristics contained in an individual review, such as the review’s rating,the style of writing, and the vocabulary used,• Features related to the reviewed item (i.e., movie) that is
the subject of the review consists of metadata such as director, movie category, and reputation such as the number ofreviews and the overall movie rating, and• Reviewer related features are features of the reviewer who
wrote the review. They consist of information about thereviewer’s experience, such as the number of reviews written
so far, and for how long they have been writing reviews, ortheir grading bias.

∗ Kakeru Ito contributed to this research while at Aoyama Gakuin University until

March 2020

2 Yahoo! Movies: https://movies.yahoo.co.jp/

1 IMDb Statistics - Press Room - IMDb: https://www.imdb.com/pressroom/stats/3 Gyao! https://gyao.yahoo.co.jp/

Conference’17, July 2017, Washington, DC, USA

Ito et al.

reviews (i.e., review targets, reviewers) and discuss the differences
and positioning of this research.
Movie Information

Link to the Streaming Site
List of User Reviews

Figure 1: Screen shot of a movie detail page in Yahoo! Movies
in Japan (Movie images were masked due to the Copyright).
Users can move to the streaming site after reading reviews

In a random forest classifier, the contribution of each feature can becalculated as the importance when performing classification. Features with high importance are expected to be a substantial factor
in determining whether the review encourages people’s moviewatching behavior. Finally, we conducted a subject experimentwith a questionnaire about reviews with these characteristics. Subjects answered how much reading a review made them want to see
the reviewed movie.
There are two practical advantages to using such a classifier for
analysis. First, it can extract reviews that are actually encouraging. Since the confidence of the classification results can rank theextracted reviews, the review site will be able to show only thereviews that are likely to be effective for decision-making. Second,features with different ranges of values and forms of expression canbe compared integrally with their contribution to the classification.
Some review-related features take real values, such as the numberof characters, while others take discrete values, such as the numberof stars. These features cannot be directly compared or analyzed.
The structure of this paper is shown below. This section described the background and purpose of our research. In section
2, related research fields with similar objectives and specification
techniques are introduced and discussed. Section 3 describes theanalysis method proposed in this research. Section 4 shows themethod and result of the actual analysis using real access logs ofYahoo! Japan and Gyao. Section 5 describes the subject experimentand its results. In section 6, we discuss the results, and in section 7,
we conclude and discuss future prospects.

2

RELATED WORKThis study identifies reviews that induce consumption behavior byusing various features. This section introduces and discusses related
studies that target reputational information or eWOM (electric
Word of Mouse). We also introduce the features associated with

2.1

Review’s Usefulness

Many studies have been conducted to determine the relationshipbetween the characteristics of a review and its usefulness.
Zhou et al. [22] define the usefulness of a review as the degree
to which reading the review improves one’s ability to evaluate theproduct. They categorize the features of reviews into two types:
numerical features and textual features. Furthermore, they discuss
the hypothesis that the two types of features influence each other.
As numerical features, they used metadata that the reader canobtain before reading the text, such as the length of the text andthe rating. As textual features, they used information that can only
be obtained by reading the text, such as the text sentiment or the
average number of words in a sentence. Experiments showed that
longer reviews give readers more confidence and are more likely
to be deemed useful.
Among the numerical features of reviews, there have been manystudies on the relationship between rating and usefulness. However,conflicting conclusions have been reported. Pan et al. [18] statedthat reviews with higher scores are more likely to be perceived as
useful reviews. On the contrary, Chua et al. [6] stated that reviews
with higher ratings are less likely to be perceived as useful reviews.
In contrast to these two results, Mudambi et al. [16] showedthat there is a non-linear relationship between the usefulness ofa review and its rating. They reported that reviews with high andlow ratings are less likely to be perceived as useful, and reviews
with average ratings are most likely to be judged as useful.
Other textual features, such as the polarity of the text sentiment,
have also been studied in terms of their effects on consumer purchasing behavior [2, 5]. The inconsistent effect of text polarity on
the usefulness of a review has also been reported by Hao et al. [9] .
Hong et al. [10], in a meta-analysis of studies on the characteristics that affect the usefulness of reviews [12, 19], showed that thecomplexity of the review text and the time elapsed since the review
was posted might have a significant effect on the usefulness of thereview.
In a similar study on the usefulness of reviews, Chen et al. [3]pointed out that reviews that are rated as “Helpful” by other users
have a more decisive influence on readers’ purchase intentions.
However, it is rare for ordinary users to click the “Helpful” button.
Therefore, it is also pointed out that the indicator of evaluationfrom users is insufficient for judging usefulness [13, 21]. It has also
been pointed out that there is a bias based on the timing of theposting of reviews. A newly posted review does not have enoughtime to be evaluated by other users.

2.2

Effect of Review Target

In this study, the characteristics of the items to be reviewed arealso used for classification. For review sites, it is known that thecharacteristics of the items affect the behavior of users.
In a survey of eWOM, Nelson et al. [17] proposed a method toanalyze two types of products, search goods and experience goods.
Following this analysis, Mudambi et al. [16] mention that the type of
product affected the usefulness perceived by review readers. HuangWhat Makes a Review Encouraging: Feature Analysis ofUser Access Logs in a Large-scale Online Movie Review Site

et al. [11] found a phenomenon that sentences including subjective
impression affect the reader’s usefulness judgment, and objective
reviews are judged as useless in the case of reading reviews ofexperience goods. Luan et al. [14] conducted an experiment using
an eye-tracking device and compared the results with those of anexperiment using a questionnaire to verify the results presented
above. The results of gaze analysis showed that users’ browsing
behavior varies not only by product type but also by review content.
In this study, we analyzed movie reviews. It is difficult to judge
the content of a movie by its specifications and performance, and it
is impossible to evaluate its quality without experiencing it. Therefore, it is close to an experience good, and as in these analyses,
we include subjective impressions and other characteristics. Also,
as with eye gaze, the analysis in this study is based on the actual
behavioral logs of users, not polls or questionnaires.

Conference’17, July 2017, Washington, DC, USA

Table 1: List of features in the review feature vector
Type

Hyposesis

FeatureReview itself

Amount of information

Text length
Title length
Spoiler tag# of words per sentence
Frequency of line breaks
Impression tags
Topic
Overall grading
Rating by viewpoint
View count# helpful

1
1
1
1
1
20
50
1
5
1
1

Running time
Release date
Title length
Category# of director’s films
Rating (age reccomendation)
Total # of reviews# fav# viewed
Total rating

1
1
1
15
1
1
1
1
1
1

# review posted
Total # of characters# of movies watched# of helpful clicked
Average # of characters% of reviews with spoiler tags
Average view count
Grading variance by viewpoint
Average of tags attached# helpful received

1
1
1
1
1
1
1
1
1
1

Readability
Review content

Popurality
Item

Metadata

Item content

2.3

Reviewer’s Authority

In addition to the reviews themselves, the impact of the reviewer
on the reviews’ usefulness has been discussed. Connors et al. [7]argued that when a reader judges the usefulness of reviews, a review
that suggests it is written by an expert tends to be judged as more
useful.
Studies have also been conducted on the relationship between
a review’s usefulness and the extent to which reviewers disclose
their profiles. The degree of disclosure and usefulness are known
to be positively correlated [1, 8]. As an example, Cheung et al. [4]analyzed what kind of reviewers are trusted through a large-scale
survey. Depending on the data set used in the study, the amount of
profile information a reviewer can disclose varies.
Different services have different forms that can be filled out for
profiles, and in some cases, there are completely anonymous review
sites. In our study, which uses the Yahoo! Movies dataset, reviewers
cannot disclose much profile information. The readers can only
see review history, screen names, and such systematic information.
Thus, different review sites may have different opportunities andtendencies to judge the authority of reviewers.

Popularity

Reputation
Reviewer

Informativeness
Authority

Total # dimension

ANALYSIS METHOD

Our method trained a random forest classifier that estimates whethera certain review induces a person to take action. This analysis
method can be used for general review sites, but we will explain
the method using movies as an example since we actually analyzed
movie reviews.
Three kinds of features can roughly characterize a certain review; the information contained in a specific review, information
about the author who wrote the review, and information about
the movie that is the target of the review. The proposed method
transforms a given review into a feature vector of 149 dimensions.
Then, the number of people who actually watched the movie rightafter reading the review was aggregated and assigned as the correct answer label. Nowadays, most review sites include a link fromthe movie review page to the viewing page of an affiliated online
streaming site. The method measures the click-through rate of thislink, and extracts the reviews where an elevated ratio of readers
moved to the streaming site. We used them as encouraging reviews,
i.e., as positive examples for the training. Reviews in this dataset

149

were classified by random forest classifier, and the contribution ofeach feature or set of features is analyzed.

3.1
3

Enthusiasm

Features of Review Itself

The first type of feature concerns the content of the review itself.
For example, the writing quality of the review itself is one of theessential indicators of whether the review is useful for decisionmaking. Therefore, we characterize the reviews by their amount of
information, readability, content, and popularity. We hypothesize
about each of these factors and represent them as 115-dimensionalfeature values.
We focused on the amount of information contained in the review. We used the number of characters in the review, the numberof characters in its title, and the spoiler flag as features focusing on
the information content of the review. The longer the review, themore content it has. It is more likely to contain helpful descriptions.
The same goes for reviews with long review titles. The spoiler flagis 0 or 1, indicating whether the review contains the story’s core.
Reviewers can arbitrarily set this flag when they submit a review.
The website does not display the text for posts containing spoilers
until the viewer clicks on it.
We considered the readability of the review; an easy-to-readreview must help readers make their decision [22]. Therefore, weConference’17, July 2017, Washington, DC, USA

Ito et al.

Online Review Site

Access Log
It is my best movie ever !!!

Roman Holiday
AVG. ★★★★★

Raw data

Review: 563 Reviews
Tag: cute, romantic, funny

Overall Grade: ★★★★★

Grade for Viewpoints

Story
Cast

Directed by: William Wyler
Release date: Aug. 27, 1953
Running time: 118 min
Category: Love Story

Direction

Movie InfoMusic

VisualOverall Grade: ★★★★★

ID: x12345

It is my best movie ever !!!

Grade for Viewpoints

Story

Proﬁle: I am a movie fan!
Watch List: 23 movies
Total Helpful: 15

Text:
A masterpiece. A
cute princess and acool newspaper
reporter. The lastscene is very sad.

Cast

Direction

VisualText:
A masterpiece. A
cute princess and acool newspaper
reporter. The lastscene is very sad.

Link: Click here to see this movie

Review site

Review History:

Roman Holiday ★★★★★
Shane ★★★★★
Casablanca ★★★★★

Review

Music

Reviewer Info
Streaming site

FeatureVector24 dimensions

10 dimensions

115 dimensions
･･･

･･･

1 dimension･･･

0 or 1Figure 2: Outline of our analysis method. We created dataset consist of 149 dimensional feature vectors of the reviews, andthe answer label that represent the encouragingness of a review.
used the length of a sentence (i.e., the average number of words ina sentence, separated by periods or exclamation marks), and thenumber of linebreaks as readability indicators.
To represent the content of the review, we used four features: The
impression tag, the topic of the words in the review text, the overallrating, and the rating by individual perspectives. The impression
tag is a function that allows reviewers to add tags when they post areview. Specifically, reviewers can select zero or more tags from 20
types. Tags contain both impressions and movie categories, suchas tearful, fantasy, and cool. Impression tags were used as binaryfeatures of 20 dimensions.
To indicate what words are used in the text of the review, weused LDA (Latent Dirichlet Allocation) to transform the words inthe text into topics consisting of 50 dimensions. LDA is a method
based on topic models and is commonly used for dimensionallycompression of document vectors. As a training corpus for LDA,
we used all review texts from our dataset.
Essentially, in LDA, a single document belongs probabilisticallyto multiple topics. This set of features accounted for more than onethird of the entire feature vector size in this experiment. Throughour preliminary experiments, we found that when each of the 50
dimensions took real values, the classifier tended to be affected by
the specific dimension in the topic, and the accuracy decreased. For
this reason, we assumed that a review sentence belongs to a single
topic and transformed the review content into a one-hot vector.
The rating of a review is also an important characteristic of thecontent of the review. In this review site, the rating is represented
as the number of stars (i.e., from one star to five stars). A review
with five stars and a review with three stars are considered to havedifferent degrees of usability for decision making. Some of the review sites allow reviewers to give a movie a score from individual
viewpoints in addition to the overall score. Our dataset has five
perspectives for a movie: story, cast, direction, visuals, and music.

Reviewers are giving a rating on a five-point scale for each perspective. These overall and point-of-view scores were treated asfeatures with one dimension each.
In addition, we also consider how popular the review is and how
much it has been viewed. Readers can also vote for a review bypressing the “helpful” button when they find the review useful. In
many studies, this vote has been treated as an essential indicator.
Review sites have a view count that indicates how difficult thereview was to read. View count indicates the amount of interest
from non-explicit readers. These values are also included as features
since they are essential in determining whether readers believe thereview.

3.2

Features of Item

The information about the item reviewed, i.e., the movie, is also
expected to affect readers’ decision-making. How seriously people
read a review may depend on the movie’s popularity. The type ofreview preferred may also differ depending on the genre and thenature of the movie. For example, a concise review will make thereader want to see an action movie. We used the movie’s metadata,
content, popularity, and reputation as features to represent a movie.
We used running time, release year, and length of title as features
to represent general information about a movie. Running time is
the movie’s length; most movies are about two hours long. The
release year was normalized from 0 to 1, since it is a value witha range of about 100 years from 1900 onwards. The length of themovie title was also used as a feature. We suspect that a short andconcise title may increase the need for a review because the content
cannot be predicted from the title.
Information about the content and quality of the movie is also
important. The category tag assigned to a movie is the most straightforward information about the movie’s content. In our dataset, there
are 15 movie categories: Sci-Fi, Fantasy, Action, Adventure, Animation, and so on. Some of these tags represent a story, and others

What Makes a Review Encouraging: Feature Analysis ofUser Access Logs in a Large-scale Online Movie Review Site

represent a technique. Therefore, a single movie often has multiple
tags. The category information was expressed as binary features in
15 dimensions.
We additionally used information about the director of the movie.
For a movie taken by a veteran filmmaker, the quality and content
may be predicted without reading a review. Therefore, we included
the total number of movies that the director took in the features. The
movie rating is another feature we used to represent the content of amovie. In general, there is a recommended age range for each movie.
In Japan, films are classified into four categories: G (for a generalaudience), PG-12 (parental guidance recommended for children
under 12), R15+ (Restricted for under 15), and R18+ (Restricted forunder 18). We used it as a 1-dimensional feature taking values one
to four.
The popularity of the movie is also an important factor. We
used the total number of reviews posted for the movie as a simple
popularity measure. The review site we used for analysis allows
users to list the movies that they saw, and movies that they want tosee, without writing a review. We used these numbers as anotherpopularity measure.
The overall reputation is also considered to contribute to watching the movie. Therefore, we averaged the review ratings of all
reviewers for the movie as a 1-dimensional feature to represent the
reputation.

3.3

Features of Reviewer

The reviewer who posted the review is also an important pieceof information in determining whether the review is useful for
decision-making [7]. The reviewer’s profile and past posting tendencies are related to the quality of the review. In addition, thereviewer’s information may influence whether readers believe thereview or not. For example, some reviewers may write incendiary reviews that spur people to action, even though they use thesame vocabulary. There may be reviewers with a large fan base(e.g., professional critics who blend into amateurs on the reviewsite). Therefore, we characterize the reviewers into 10-dimensionalfeatures by their enthusiasm, the informativeness of their posts,
and their authority.
Note that, in this analysis, we did not use reviewers’ personal
information such as age and gender, but only characteristics related
to the review behavior itself. This is because supplying personal
information is optional on many review sites, and the reliabilityof this information cannot be verified. In addition, by using only
public review information but not personal information, our model
can be used with a wider range of review data.
As the first aspect of a reviewer’s characteristics, we focused on
their enthusiasm; how much the reviewer has actively contributed
to the review site. It has been pointed out that the amount of review
experience of a reviewer is related to his or her review ability [15].
As specific features, we used the total number of reviews and thetotal number of characters as the amount of information in thesubmitted reviews. On the review sites we analyzed, apart from thenumber of reviews written, the number of movies the reviewer has
seen can be posted as a movie viewing record. Also, if the reviewer
has viewed many reviews, then that reviewer has a high interestin this review site. Therefore, we included the number of times a

Conference’17, July 2017, Washington, DC, USAreviewer clicked the “helpful” button for someone else’s review inthe features.
As for the other characteristics of the reviewers, we focused on
informativeness that represents how much information each reviewof that reviewer had. Specifically, the ratio of reviews with spoiler
tags among all reviews and the average number of words in thereviews were used as features. Reviewers can add spoiler tags toeach review they write. Reviewers who put a lot of spoiler tags forhis or her reviews are more likely to have actually watched themovie all the way through, and mentioned the movie’s scenario.
Finally, we quantify the reviewer’s authority, i.e., how much thereviewer knows about the movie. For this purpose, we used thegrading variance for each viewpoint, the number of impression
tags, and the average of the usefulness they received. The varianceof rating by viewpoint was used to indicate the reviewer’s familiarity with the movie [20]. Some reviewers may focus on only one
aspect of a movie to evaluate it. For example, some reviewers may
be interested only in the music. Reviews by such reviewers will,
for better or worse, influence the reader’s decision-making. The
variance of the average ratings for each viewpoint was used as anindicator of the reviewer’s concern with the viewpoint.

4

ANALYSIS USING REAL DATAIn this study, we used the review data of Yahoo! Movies and theaccess log of Yahoo! Movies to identify the reviews that contributeto decision-making. This chapter describes the details of the dataset,the actual analysis procedure, and the results.

4.1

Log Data Collection and Cleansing

The dataset used consists of review data from Yahoo! Movies, access logs to each review in Yahoo! Movies, and access logs to thestreaming page in GYAO!. The logs were collected over a period of
61 days, from November 1, 2019, onwards and before December 31,
2019.
The access log of the review site consisted of user, movie, review,
and visit date and time. This is described for all visitors who accessed the movie during the period. The movie viewing log consistsof the user, the movie, and the date and time of the visit. Users visit
Gyao! not only from Yahoo! Movies but also from various othersites such as search engines. Therefore, we extracted only the logs
where the user with the same user ID actually accessed both sites
in a short period of time.
For data cleansing on a per-user basis, we excluded the behavior
of users who had never watched a movie on Gyao!. This excludes
users who watch movies on other movie streaming sites and users
who do not watch movies on the Internet. By limiting the numberof users, we can analyze the difference between the actions thatencouraged Gyao! users to watch and those that did not.
Finally, we randomly selected 70,000 samples (i.e., a suitableamount for handling and cross-validation). Note that the numberof users who moved from Yahoo! Movies to Gyao! is very low
compared to the number of users and accesses to these services.
The use of review sites is diverse. Some users want to know the
reputation of a movie they have seen, and some of the others read
reviews to purchase a DVD. Some users watch movies on streamingsites other than Gyao!, even if the logs of users who have never usedConference’17, July 2017, Washington, DC, USA

Ito et al.

Table 3: Importance for each set of features.

Table 2: Performance of the classifier and variants that use
limited features
Features𝐹 1 score

Review itself + reviewer + item
Review itself + item
Review itself only

0.771
0.786
0.568

Type

Hypothesis

Review itself(Total: 0.305)

Amount of information
Readability
Review content
Popurality

0.056
0.048
0.137
0.064

Item
(Total: 0.425)

Metadata
Item content
Popurality
Reputation

0.039
0.117
0.210
0.059

Reviewer(Total: 0.273)

Enthusiasm
Informativeness
Authority

0.138
0.052
0.083

Gyao! were excluded. In this experiment, for data collection reasons,
we had to consider that these behaviors were not encouraged.

4.2

Labeling Encouragingness

We defined the encouragingness score 𝑒𝑛𝑐 (𝑟𝑘 ) as the ratio of readers
who moved to the streaming site after reading a certain review 𝑟𝑘
to all readers who read review 𝑟𝑘 . The 𝑘-th review is expressedas 𝑟𝑘 in 𝑅, where 𝑅 is the set of all reviews in the dataset. The
encouragingness score 𝑒𝑛𝑐 (𝑟𝑘 ) is defined as|𝑈 𝑤𝑎𝑡𝑐ℎ (𝑟𝑘 )||𝑈(𝑟 )|− Í 𝑟𝑒𝑎𝑑 𝑘𝑟 𝑝 ∈𝑅 |𝑈 𝑤𝑎𝑡𝑐ℎ (𝑟 𝑝 )|𝑟 𝑝 ∈𝑅 |𝑈𝑟𝑒𝑎𝑑 (𝑟 𝑝 )|𝑒𝑛𝑐 (𝑟𝑘 ) = Í(1)where |𝑈𝑟𝑒𝑎𝑑 (𝑟𝑘 )| is the number of users who read the review 𝑟𝑘 ,and |𝑈 𝑤𝑎𝑡𝑐ℎ (𝑟𝑖 )| is the number of users who access the streamingsite after they read the review 𝑟𝑘 . Since 𝑈 𝑤𝑎𝑡𝑐ℎ (𝑟𝑘 ) is a subset of
𝑈𝑟𝑒𝑎𝑑 (𝑟𝑘 ), 𝑒𝑛𝑐𝑟𝑘 falls between 0 and 1.
The value of 𝑒𝑛𝑐 (𝑟𝑘 ) is continuous. There are two ways to use
such a value as a correct answer label. One way is to set a threshold
and divide it to 0 or 1. The other way is to split using a fixed
percentile. In this analysis, we consider the top ten percent as
correct answers and the other 90 percent as negative examples. For
the random forest training, the same number of samples were usedfrom correct and negative examples.

4.3

Feature Analysis by Classification and
Cross-Validation

We classified these reviews into encouraging reviews and otherreviews using random forest. As the implementation of random
forest classifier, we used the existing machine learning library calledscikit-learn4 . First, we made two variant methods; one uses only
features of the review itself, while the other uses review and itemfeatures. The performance of each method was calculated by fivefold cross-validation. Classification and cross-validation were doneusing the initial parameters of scikit-learn. The number of trees is300, and the maximum depth of a tree is 10.
The classification result is shown in table 2. The best performingclassifier was the one that used the features of the review itself andthe item, with an 𝐹 1 score of 0.786. On the contrary, the model using
all the features showed decreased accuracy. This indicates that thefeatures associated with the movie have a significant impact on theclassification performance of the review’s encouragingness.
Next, the importance of each feature of the classifier using all thefeatures was tabulated for each element of interest. In scikit-learn,
which was used in this experiment, the importance of a feature ina random forest is the average of the importance of the nodes of
4 scikit-learn:https://scikit-learn.org/

Total

Importance

1.000

each decision tree that composes the forest for each feature. In arandom forest, multiple short decision trees are created, and not
all features are necessarily used in all trees. Let 𝑓𝑖 be the feature
value of the 𝑖-th dimension of the feature vector. For a node 𝑛𝑖 to
be classified by feature 𝑓𝑖 in a tree 𝑡 𝑗 , the probability of reaching
it is expressed as 𝑝 (𝑛𝑖 ) (this probability will be used as a weight).
Since the decision tree is a binary tree, the child nodes connected
to a node 𝑛𝑖 are divided into right and left, and denoted as (𝑛𝑖𝑟 )and 𝑛𝑖𝑙 . The importance 𝑖𝑚𝑝 tree (𝑛𝑖 , 𝑡 𝑗 ) of feature 𝑛𝑖 in tree 𝑡 𝑗 canbe represented as
𝑖𝑚𝑝 tree (𝑛𝑖 , 𝑡 𝑗 ) = 𝑝 (𝑛𝑖 )𝑔(𝑛 𝑗 ) − 𝑝 (𝑛𝑖𝑟 )𝑔(𝑛𝑖𝑟 ) − 𝑝 (𝑛𝑖𝑟 )𝑔(𝑛𝑖𝑟 )(2)where impurity is represented by 𝑔(𝑛𝑖 ). For all the trees 𝑇 used inthat random forest classifier, the importance of a feature 𝑓𝑖 is
1 Õ
𝑖𝑚𝑝 tree (𝑛𝑖 , 𝑡 𝑗 ),(3)
𝑖𝑚𝑝 forest (𝑓𝑖 ) =|𝑇 |𝑡 𝑗 ∈𝑇i.e., the average of the importance of the feature in all trees.
The importance of each set of features is summarized in Table
3. In terms of the contribution of each set of features, features associated with the movie have the highest importance. The total
importance of the item features was 0.42. The importance of the review content was the next highest at 0.30, and the reviewer-related
features were the lowest at 0.27. From the most important items,
we were able to deduce that the movie’s popularity is especiallyencouraged by the readers’ movie-watching behavior. The otherimportant features were the reviewer’s enthusiasm and the content
of the review.

5

SUBJECT EXPERIMENTIn order to confirm whether the findings obtained by the feature
analysis are useful in the real world, we conducted a subject experiment. In this experiment, we check whether the classification
results by Random Forest differ from the actual human feeling. For
this purpose, we collected reviews that were not used in training(i.e., reviews for movies that were not distributed on the streamingsite), and classified them by whether they encourage readers towatch the reviewed movie. Experiment participants were asked in a

What Makes a Review Encouraging: Feature Analysis ofUser Access Logs in a Large-scale Online Movie Review Sitequestionnaire whether reading the review had actually made themwant to see the movie.

5.1

Experimental Setting

We conducted a questionnaire-based study to analyze whether thereviews that were determined by the classifier to be encouraging ofmovie-watching behavior are actually considered encouraging toparticipants. For the experiment, we selected the ten movies withthe highest box-office revenue in Japan in 2019. We used the trainedclassifier to estimate the encouragingness of all the reviews posted
for each of these movies. We sampled the reviews by the confidence
score of the random forest classification: top five and bottom five.
The participants in the experiment were six university students.
The reviews extracted by the classifier were presented for participants in random order. Participants were asked to indicate for eachreview whether they felt encouraged to watch that movie whenreading the review. They were asked to rate each review on a fourpoint scale, from 1 (not encouraged) to 4 (well encouraged). The
review was presented in a stand-alone text format. In other words,
no metadata about the film and no information about the review’s
author was provided.
We compared the classification results of four methods: the proposed method that includes all features, the variant method thatuses item information and reviews, the method that uses only reviews itself, and a baseline method. The baseline method randomly
selects and ranks ten reviews from the entire set of reviews of amovie without using any classifier.

5.2

ResultThe score assigned by humans for each method is shown in Table 4.
Reviews estimated to be encouraging by the classifiers were rated as
more facilitative for humans than other reviews. These differenceswere significant at 𝑝 < 0.01 as a result of validation by Student’s
𝑡 test. For the randomly selected reviews, there was no difference
between the top and bottom, as a matter of course.
Throughout, the average of reviews’ encouragingness score was
2.45 on our four-point scale. It suggests that most reviews on thereview site resulted in an evaluation of “slightly encouraged to see
the movie”. Focusing on the values, the method that best estimatedthe encouraging reviews was the classifier that only used featuresof the item and the review itself. The difference between the top
and bottom is also the highest at 0.88. This indicates that the classification results by this method and the degree of encouredingness
felt by the human are close. In contrast, the classifier using theauthor’s characteristics disagreed with participants.

6

DISCUSSION

Overall, we were able to estimate whether the reader would actually go to the streaming site after reading the review by using thefeatures surrounding the review. Through the experiment where
we show the reviews estimated encouraging to the participants,
the classification results and the participants’ opinions generally
agreed. These results suggest that our method is able to correctly
extract reviews that encourage people’s viewing behavior.
Let us discuss the features of the movies that were judged tobe the most effective. From the comparison of classifiers (see table

Conference’17, July 2017, Washington, DC, USA

2) and the importance of the features (see table 3), it was foundthat the features associated with the movie contributed greatly
to the estimation of whether or not the movie encouraged moviewatching behavior. This indicates that whether a person wants towatch a movie is more influenced by the reviewed movie itself thanby the text of the review.
Among the features of the movie, the popularity of the movie
had the greatest effect. In other words, if a movie is popular, users
will feel compelled to watch that movie regardless of the content
of the reviews. The popularity of a movie has a larger impact thanother movie features (e.g., content, metadata). It is possible thatpeople judge whether to see a movie by its popularity, not by its
content. Among the features associated with a movie, the reputation
information of a movie has a relatively small impact. Even if a movie
is heavily criticized in reviews, people may still watch it if it is afamous movie.
Here we need to consider the way movie review sites are used.
People may first decide which movie to watch and then go to thereview site to check the reviews. Or, they may somehow find themovie they want to see by looking at reviews for various movies
and then go to the streaming site if there is a link. Since the usage ofreview sites is different for each person, additional analysis which
focuses on individual users is needed in the future.
Next, we discuss the features of the review contents. Even witha classifier that uses only the features inherent in the review forclassification, we were able to extract reviews that entailed viewing
with an accuracy of 𝐹 = 0.57. From the importance of the features
shown in table 1, the features related to review content had thegreatest effect for encouraging movie-watching. The features withthe highest importance were the grades of each viewpoint, followedby the topic of the review text. This indicates that users may pay
attention to the viewpoint when reading reviews. For example, auser who is interested in music might look at the grades related tomusic, and then read the sentences related to music in the review
text.
In addition, we found that the amount of information in the review did not have much effect on movie viewing behavior. From thisresult, we can deduce some tips for writing reviews that encourage
people. That is, it is futile to spend the effort to write lengthy, informative reviews. Reviewers should pay attention to the viewpoints,
and review the film from each viewpoint.
Finally, we also found that there is still room for improvementin the method. The label of correct answers seems unnatural. The
negative examples in the behavioral logs may include people whohave already seen the movie or users of another streaming site. In
addition, the dataset contained movies that were seen by many people and movies that only a few people saw. These factors may givesome biases to our dataset. The dataset itself needs to be analyzed
and cleansed. We analyzed only two months’ worth of logs (in consideration of privacy protection). Therefore, there is a possibility
that major movies that are released during that period may haveaffected the results. Long-term analysis is desired in the future.

7

CONCLUSION

Using a large-scale access log of an actual online movie review
site, we analyzed the characteristics of the reviews that encouragedConference’17, July 2017, Washington, DC, USA

Ito et al.

Table 4: Result of the subject experiment if the review encourages the participant to watch the reviewed movie released in2019 (* means the statistically significant difference of 𝑡 test; * 𝑝 < 0.05, ** 𝑝 < 0.01)
title

All features
Top 5 Bottom 5

Review itself + item
Top 5 Bottom 5

Review itself only
Top 5 bottom 5

RandomTop 5 Bottom 5

Kingdom
Toy Story 4
Detective ConanONE PIECE: STAMPEDE
Weathering with You
Avengers: Endgame
Aladdin (2019)
Star Wars Ep. 9
Frozen II
The Lion King (2019)

2.90*
2.90
1.90
2.70**
2.50**
2.40
3.10**
2.10
2.70*
2.20

1.70
2.70
2.10
1.90
1.80
2.00
1.40
1.70
1.60
2.60

3.50**
1.40
2.60*
3.20**
2.80
2.50*
3.20*
3.20**
3.00
3.00

1.20
2.40
1.50
1.70
2.40
1.50
2.20
1.70
2.60
2.40

3.40*
1.70
2.60*
3.10*
3.60**
2.40
2.60
2.60
3.30
3.00*

2.50
1.60
1.40
2.20
2.10
2.20
3.00
2.50
3.00
2.00

2.90
1.40
2.20
3.10
3.50
2.20
2.50
2.10
2.90*
3.20*

2.70
1.60
1.90
2.40
3.60
2.90
2.60
2.00
2.30
2.60

average

2.54**

1.95

2.84**

1.96

2.83**

2.25

2.60

2.46

readers to watch a movie after reading a review of the movie. We
focused on the users’ page transitions from the review site to themovie streaming site after they read the review. For the analysis,
we used a random forest classifier trained to determine whether areview caused a movie-watching behavior. We conducted featureimportance-based analysis using three types of features; featuresof the review itself, features about the item, and features of the reviewer. We used the actual access log of Yahoo! Movies Japan (one of
the biggest movie review sites in Japan) and Gyao! (a movie streaming site operated by Yahoo! Japan). Through the cross-validation
experiment, the classifier was able to classify encouraging reviews.
In particular, features related to the movie’s popularity were shown
to affect the encouragement of people’s movie viewing. A subject
experiment confirmed that these features contribute to the review’s
usefulness.
The contribution of this study is that we made it possible todiscover encouraging reviews from actual access log, and that the
popularity of the movie and the author’s review experience affected
the readers’ decision-making process. As future work, we need toevaluate whether classified reviews actually encouraged users. We
will be able to conduct A/B testing-based user experiments using
our results. Application is also an important technical issue. For
example, when a reviewer posts a new review, the system could
display advice that says, “Your review will be better if you change
here .” These would be of great help in improving the quality ofreview sites.

ACKNOWLEDGMENTS
This work was supported by JSPS KAKENHI Grants Number 18K18161,
21H03775, and 18H03243.

REFERENCES[1] Einar Bjering, Lars Jaakko Havro, and Øystein Moen. 2015. An empirical investigation of self-selection bias and factors influencing review helpfulness. International
Journal of Business and Management 10, 7 (2015), 16.
[2] Don Charlett, Ron Garland, Norman Marr, et al. 1995. How damaging is negative
word of mouth. Marketing Bulletin 6, 1 (1995), 42–50.
[3] Pei-Yu Chen, Samita Dhanasobhon, and Michael Smith. 2008. All Reviews are
not created equal: The disaggregate impact of reviews on sales on Amazon. com.

In Carnegie Mellon University Working Paper.
[4] Christy MK Cheung, Matthew KO Lee, and Neil Rabjohn. 2008. The impact ofelectronic word-of-mouth: The adoption of online opinions in online customer
communities. Internet research 18, 3 (2008), 229–247.
[5] Judith A. Chevalier and Dina Mayzlin. 2006. The effect of word of mouth onsales: Online book reviews. Journal of Marketing Research 43, 3 (2006), 345–354.
[6] Alton Chua and Snehasish Banerjee. 2014. Understanding review helpfulness
as a function of reviewer reputation, review rating, and review depth. Journalof the Association for Information Science and Technology 66 (06 2014). https://doi.org/10.1002/asi.23180[7] L. Connors, S. M. Mudambi, and D. Schuff. 2011. Is It the Review or the Reviewer?
a Multi-Method Approach to Determine the Antecedents of Online Review Helpfulness. In 2011 44th Hawaii International Conference on System Sciences. 1–10.
https://doi.org/10.1109/HICSS.2011.260[8] Chris Forman, Anindya Ghose, and Batia Wiesenfeld. 2008. Examining therelationship between reviews and sales: The role of reviewer identity disclosurein electronic markets. Information systems research 19, 3 (2008), 291–313.
[9] Yuanyuan Hao, Qiang Ye, Yijun Li, and Zhuo Cheng. 2010. How Does the Valenceof Online Consumer Reviews Matter in Consumer Decision Making? Differences
between Search Goods and Experience Goods. 1–10. https://doi.org/10.1109/
HICSS.2010.455[10] Hong Hong, Di Xu, G Alan Wang, and Weiguo Fan. 2017. Understanding the determinants of online review helpfulness: A meta-analytic investigation. Decision
Support Systems 102 (2017), 1–11.
[11] Liqiang Huang, Chuan-Hoo Tan, Weiling Ke, and Kwok-Kee Wei. 2013. Comprehension and assessment of product reviews: A review-product congruity
proposition. Journal of Management Information Systems 30, 3 (2013), 311–343.
[12] Zhiwei Liu and Sangwon Park. 2015. What makes a useful online review?
Implication for travel product websites. Tourism Management 47 (2015), 140–
151.
[13] Yue Lu, Panayiotis Tsaparas, Alexandros Ntoulas, and Livia Polanyi. 2010. Exploiting social context for review quality prediction. In Proceedings of the 19thInternational Conference on World Wide Web. 691–700.
[14] Jing Luan, Zhong Yao, FuTao Zhao, and Hao Liu. 2016. Search product andexperience product online reviews: An eye-tracking study on consumers’ review
search behavior. Computers in Human Behavior 65 (2016), 420 – 430. https://doi.org/10.1016/j.chb.2016.08.037[15] Julian John McAuley and Jure Leskovec. 2013. From amateurs to connoisseurs:modeling the evolution of user expertise through online reviews. In Proceedingsof the 22nd international conference on World Wide Web. 897–908.
[16] Susan M Mudambi and David Schuff. 2010. What makes a helpful review? A
study of customer reviews on Amazon. com. MIS quarterly 34, 1 (2010), 185–200.
[17] Phillip Nelson. 1974. Advertising as information. Journal of political economy 82,
4 (1974), 729–754.
[18] Yue Pan and Jason Q Zhang. 2011. Born unequal: a study of the helpfulness ofuser-generated product reviews. Journal of Retailing 87, 4 (2011), 598–612.
[19] Pradeep Racherla and Wesley Friske. 2012. Perceived ‘usefulness’ of online
consumer reviews: An exploratory investigation across three services categories.
Electronic Commerce Research and Applications 11, 6 (2012), 548–559.
[20] Yoshiyuki Shoji, Makoto P. Kato, and Katsumi Tanaka. 2014. Can Diversity
Improve Credibility of User Review Data? Springer International Publishing,What Makes a Review Encouraging: Feature Analysis ofUser Access Logs in a Large-scale Online Movie Review Site

Cham, 244–258. https://doi.org/10.1007/978-3-319-13734-6_17[21] Jiliang Tang, Huiji Gao, Xia Hu, and Huan Liu. 2013. Context-aware review
helpfulness rating prediction. In Proceedings of the 7th ACM Conference on Recommender Systems. 1–8.

Conference’17, July 2017, Washington, DC, USA[22] Y. Zhou and S. Yang. 2019. Roles of Review Numerical and Textual Characteristics
on Review Helpfulness Across Three Different Types of Reviews. IEEE Access 7(2019), 27769–27780. https://doi.org/10.1109/ACCESS.2019.2901472