Doc2Vec-based Approach for Extracting Diverse
Evaluation Expressions from Online Review Data
Kosuke Kurihara∗

Yoshiyuki Shoji

Yahoo JAPAN Corporation
Tokyo, Japankurihara@sw.it.aoyama.ac.jp

Aoyama Gakuin University
Kanagawa, Japanshoji@it.aoyama.ac.jp

Sumio Fujita

Martin J. Dürst

Yahoo JAPAN Corporation
Tokyo, Japansufujita@yahoo-corp.jp

Aoyama Gakuin University
Kanagawa, Japanduerst@it.aoyama.ac.jp

ABSTRACTThis paper proposes a method for extracting diverse expressions
from online movie review texts for a given keyword query. Whenpeople watch a movie that makes them cry, they generally do not say“I cried.” Instead, they use such euphemistic language as “I needed ahandkerchief” or “My makeup was running.” To enable information
retrieval based on audience reactions such as “movies that make mecry” using review texts, a variety of paraphrased expressions must
be collected for arbitrary queries. Our proposed method extracts
such expressions from review datasets by applying two extensions
to Doc2Vec: 1) it changes the granularity of the training sentences
to mitigate a lack of context, and 2) it applies query expansion
for similarity calculation in advance. We conducted a large-scale
experiment using crowdsourcing with 1.29 million actual sentences
taken from Yahoo! Movies, Japan. The experimental result revealed
that changing the training data granularity and adding the query
expansion are both effective to accurately collect more diverse
expressions that have a meaning similar to the given query.

CCS CONCEPTS• Information systems → Document topic models.

KEYWORDS
online review, doc2vec, euphemism

1

INTRODUCTION

As internet broadband and smart device proliferation deepen, people are streaming and watching more and more movies online.
Searching for movies to stream has become a part of our lives.
Therefore, finding a movie that matches our information needs is adifficult information retrieval problem.
For example, if a person wants to watch a tear-jerker movie thatmakes him/her cry, using a text search at a movie information site
is a simple way. Such text search systems look for movies whose
titles or descriptions contain the term related to the query keyword.
However, a movie that is causing crying rarely contains the word
“tear-jerker” in its title. The descriptions are mostly comprised ofsynopses; the word “tear-jerker” rarely appears in the synopsis. A∗ Kosuke Kurihara contributed to this research while at Aoyama Gakuin University

until March 2021.

simple text search is probably an insufficient way to find a movie
to watch.
One cause is the problem that the metadata of a movie might not
represent its audience’s impressions. Some movie information sites
assign tags to solve this problem. Tags are concise and probably
reflect the impressions of audiences. However, the coverage of tags
is inadequate to meet all information needs.
Users often want movies that match trivial and personal information needs. For instance, for such unusual information needs as“movies that make you want to go on a trip” no corresponding tag
exists. The information granularity of tags is rougher than keyword
queries.
Recommendations are another approach for identifying a movie.
Most online video streaming sites recommend movies based on
personal viewing history. However, in such recommendation systems, users cannot directly input their interests or desires. Viewing
history is too vague to reflect complex needs.
Online review sites often compensate for such deficiencies. Reviews on such sites contain information about movies written by
many different people. Reviews include how they felt after watching a particular movie as well as its notable features. Since they
contain more information about a movie than metadata or tags,
they can be easily matched with queries because of their flexible
descriptions, high coverage, and fine granularity.
Unfortunately, at the moment, such reviews are being underutilized. To learn about movies based on reviews, we have to lookindividually at each review. Reading each and every review is timeconsuming and increases the risk of being exposed to spoilers. In
the current situation, although reviews are suitable for determining
whether a certain movie will make one cry, they cannot be used tofind a list of movies that will make you cry. Thus, a method must be
established that allows users to search for a sad movie by inputting
the keyword query “cry.”Writing flexibility is another factor that restricts such searches.
People sometimes use euphemisms and metaphors to describe theiropinions. When people watch a tear-jerker movie that makes themcry, they might not clearly admit “it is a tear-jerker movie.” Instead,they use such euphemistic language as “I needed a handkerchief”
or “My makeup was running.” A search algorithm needs to matcha query with a corresponding evaluation expression.
To solve this problem, we propose a method that matches a certain keyword query with expressions in reviews using Doc2Vec [13],Conference’17, July 2017, Washington, DC, USA

which is a very popular method that vectorizes sentences and calculates the similarity between sentences and queries. However,
Doc2Vec performs with low accuracy when learning short sentences because they do not provide enough context. Each evaluation
expression included in the review is short. Therefore, for example,
it is impossible to compare the keyword query “nostalgic” withsuch short phrases as “brings back memories” or “reminds me of
my childhood” with sufficient accuracy.
Our method, which ingeniously uses Doc2Vec to extract morediverse evaluation expressions from the real data of movie reviews,
contains two improvements:• it changes the granularity of the training sentences to compensate for the lack of context, and• it applies query expansion for similarity calculation in advance.
First, when training the Doc2Vec model, the method modifies
the data granularity with the idea of target-topics. This expansion supplements the lack of context during the training phase of
Doc2Vec when learning short sentences. We introduce the term
“target-topic” for the objects or the referents of social media posts.
Posts on social sites (e.g., movie review sites, and online forums)
generally have a target-topic. For instance, a social media post often
contains one or more hashtags, a review in an online review site
has a target item, and a comment in an online discussion forum has
a news article as the topic. We modify Doc2Vec using target-topics
as an additional context for training Doc2Vec networks. We expectthat this step will allow the similarity to be exploited between sentences that are related to the same target-topic and improve search
accuracy.
For the second improvement, the method uses query expansion
techniques before vectorization for similarity calculation. During
the actual search, the algorithm calculates the similarity between
the vectors of the query and the respective evaluation expressions.
The query and the evaluation representation differ in length andinformation content. Therefore, before vectorizing the query, we
use Word2Vec for query expansion. A keyword query consistingof only one or a few words is made into a short bag-of-words
consisting of synonyms, aligning the granularity of the query andthe evaluation expression to improve the vectorization accuracy.
We conducted a large-scale evaluation experiment to confirm
the effect of both extensions. We compared the accuracy of thesearch result rankings of the six methods (i.e., four variant methods
comprised of a combination of the two proposed improvements
and two baselines) using a large review dataset consisting of over
60,000 movies on an actual movie review site. Each search result
was labeled by crowdsourcing through over 20,000 tasks.
This research is a direct sequel to our short paper[11]. This paper
is structured as follows. In this section, we explained the motivationand the goal of this research. Section 2 introduces existing researchon reputation mining, review analysis, and information retrieval
using distributed representation. In Section 3, we describe the details of our method proposed in this study. Section 4 shows thesettings for the experimental evaluation and its results. Section 6
discusses the results obtained through the experiments, and Section
7 concludes with experimental results and explains future work.

Kurihara et al.

2

RELATED WORKThis section introduces related work from the viewpoint of our
proposed method and application. This research uses distributed
representation to extract reputations from online review sites andmake them searchable. We discuss research on reputation mining,
research related to review information, and research on information
retrieval using distributed representation.

2.1

Opinion and Reputation Mining

Opinion mining or reputation extraction, which estimates an item’s
reputation from social sites, has been widely studied [4]. Electronic
Word-of-Mouth (eWOM) is a critical information source that canchange people’s purchasing behavior. The most classic reputation
analysis methods extracted the overall sentiment of articles abouta certain product. The extracted sentiments can help users choose
based on more positive or negative articles about a particular product. As a more advanced method, extracting aspects from documents about an item and estimating the polarity for each aspect
is becoming more common [6]. For example, it is possible to learn
the features of a product for individual aspects, such as “screen
size, rated high” and “picture quality, rated low” for a certain television [18]. In particular, many studies have extracted sentimentand polarity toward products [14]. In recent years, machine learning methods have been widely used for these purposes [10, 19].
For example, Titov et al. [22] proposed a method to extract andsummarize word-of-mouth perspectives using a Bayesian model.

2.2

Online Review Analysis

Online review analysis is the second research field strongly related
to our proposal. Online reviews are a powerful information resource
for item retrieval [9], recommendations [25], decision support [7,27], and so on. Singh et al. [20] and Bader et al. [1] focused on
expressions and sentiments in reviews. Jo et al. [8] proposed amethod that automatically detects a combination of various aspects
and polarities in reviews. Tan et al. [21] proposed another way tofind short sentences that have a similar sentiment. To search formovies with arbitrary keywords, the polarity must be computed for
an infinite number of aspects. For example, it is difficult to predict
and calculate in advance the aspect of the “degree to which viewing
makes you hungry.” Therefore, we need a text search-like methodthat can execute on demand. One of the benefits of our researchis that it can also search for sentences with any aspect, not only
sentiments but also story patterns or genres.
In recent years, machine learning [3] and ontologies [26] havebecome more common as methods for handling movie review information. Such technologies need to prepare a sentiment label
dictionary in advance. It is difficult to make movies searchable byarbitrary keywords using these techniques.
Another common approach is to summarize movie reviews to
help users choose suitable movies to watch next. Zhuang et al. [27]proposed a method for summarizing movie reputations by applying classical opinion analysis techniques to movie reviews. Liu etal. [15] summarized movie reviews for mobile devices using Latent
Semantic Analysis (LSA). Although these methods can determine amovie’s reputation, they cannot search for movies by keywords.

Doc2Vec-based Approach for Extracting Diverse
Evaluation Expressions from Online Review Data

Online Review Site (I, R, S, T)
Reviews

Review r11
Sentence s111
s112
Term t1111 t1112
t1121 t1122
s121
t1211 t1212

Review r12
s122
t1221

Conference’17, July 2017, Washington, DC, USA

Items(Target-Topic)

Item i1

Review rmj

Sentence smjk
Term tmjkn

Item imFigure 1: Site structure of a typical online review site. Everysentence points to one item (target-topic).

2.3

Information Retrieval Using Distributed
Expression

Our method is an example of information retrieval with distributed
expressions. Many methods use Word2Vec or Doc2Vec to find information from social sites. Gysel et al. [24] also used Doc2Vec models
for short sentences on social sites. This active research field tackles
short sentences on social sites. Trieu et al. [23] proposed a method
for tagging and classifying news information posted on Twitter andsearching for similar news. Neither uses target-topics to improvesearch accuracy. Zuo [28] et al. used external information to vectorize short sentences in social sites for probabilistic topic models. Our
method also uses the external context for Doc2Vec-based vectorization. The main difference is that we use target-topics as external
information; our method does not need ontologies or dictionaries.
Many studies use distributed representation for applications similar to our study. Barkan et al. [2] proposed Item2Vec, a distributed
representation of products in e-commerce sites. Item2Vec applies
the Skip-Gram model to infer items based on a set of simultaneously
purchased items as one item. This makes it possible to discoversimilar items. Phi et al. [17] also vectorized products in e-commercesites for collaborative filtering. Their research treats a user’s purchase history as a document for Doc2Vec learning. In this way,
both users and items can be represented in a distributed representation. In research on the distributed representation of movie
information for recommendations, Liu et al. [16] used Doc2Vec torecommend movies. Our work similarly vectorizes movies fromtheir surrounding documents, although the purpose is different
from their research, since it extracts evaluative expressions from
reviews.

3

METHOD

This section describes our method for discovering various evaluation expressions for arbitrary keywords from a large dataset inpractice. The method consists of three main parts: preprocessing,
vectorization of the representation using Doc2Vec, and similarity
calculation using query expansion.

3.1

Preprocessing

A dataset taken from an online movie review site is preprocessed for
actual computation. Since the experiment uses actual large-scale
review data written in [redacted for anonymity], our country’s
language, we also discuss problems specific to it.
The information on a review site generally consists of threelayers: site, movie, and review (Fig. 1). One site is associated withmultiple movies, and one movie is associated with multiple reviews.
Each element has a variety of metadata attached to it. A movie isgiven such information as to its title, director, actors, and year of
release. Its review is generally accompanied by such information
as an author, a grade, a tag, and a review date.
In this study, we vectorize the expressions in the review andperform the learning on a sentence-by-sentence basis in each review(i.e., 𝑠𝑚 𝑗𝑘 in Fig. 1). Since we assume that the reviews for the samemovie are contextually related, the site’s entire information is usedduring learning.
First, a sentence written in a natural language is split into words.
In the documents in Japanese, words are not divided by spaces.
Therefore, we used a morphological analyzer to split them intomorphemes.
Next we filtered the terms by word classes and rule-based cleansing. The method used nouns, adjectives, and verbs, unlike othergeneral methods using Word2Vec, which often extract only nouns.
In this application, users are likely to input how they feel and how
they do after watch the movie, and adjectives in their queries. Other
useful watching information includes emotions and impressions
when watching a movie and the suitable situations for watching
it. Therefore, we added most words to the training data, excluding
particles. Since the method uses words other than nouns, the total
amount of words and noises increased. The method uses language
patterns and rules to remove unnecessary words. First, it normalized the sentence; it conjugated all verbs into standard forms andchanged all nouns to their singular forms. All words were lemmatized, leaving just the stem. Next we removed words with extremely
few letters because words consisting of only one or two charactersin specific character types are probably noise. Most are fragmentsof colloquialisms that could not be correctly morphologically analyzed. Therefore, such words were removed. Numbers and symbols
were also removed.
Similarly, control characters, special symbols, and low-frequency
words were removed. On online review sites, a variety of users post
reviews from various devices. As a result, posts sometimes containwords in different languages or special characters, depending on
their devices and environments. For example, some posts that haveincorrect line feed codes contain too much white space to makethe review look better. Therefore, we replaced such white spaces
as consecutive spaces or tab characters with a single white space.
Control characters and line breaks were also removed.
We treated each sentence as a single evaluation expression foreach of the preprocessed sentences. In other words, each sentence
was removed with symbols to indicate the end of the sentence (i.e.,
period, exclamation mark, etc.).

Conference’17, July 2017, Washington, DC, USA

3.2

Distributed representation vector of the term tmjkn

Vectorizing Sentences Using Target-Topic
Aware Doc2Vec

Using Doc2Vec, this method next vectorizes each sentence, represented as a series of valid words obtained in the preprocessing.
Overall, our proposed method is a variant of the Paragraph Vector
Distributed Memory (PV-DM) model of Doc2Vec. First, a two-layer
network is trained by estimating the next term in a sentence from its
context. In the original PV-DM model, a separate context is used foreach sentence. We modify this approach using the target-topic itself
as the context for all the sentences about it. Second, each sentence
is vectorized using the trained network. This step is identical as in
the original model; the granularity is different for the training andvectorization steps.
We used an online review site as an example to explain thedetails of our new method; it is one of the most typical examples ofa CGM (Consumer Generated Media) site. Online review sites havea common structure, as shown in Fig. 1. In this example, we call areview’s target-topic an “item” for clarity. A review site consists ofitems (target-topics) 𝐼 , reviews 𝑅, sentences 𝑆, and terms 𝑇 . Each
item 𝑖𝑚 is discussed by many reviews. Each review 𝑟𝑚 𝑗 for item 𝑖𝑚consists of a number of sentences. Each sentence 𝑠𝑚 𝑗𝑘 is a sequenceof terms denoted by 𝑡𝑚 𝑗𝑘𝑛 . The goal of our method is to accurately
vectorize 𝑠𝑚 𝑗𝑘 . For that purpose, our method uses item 𝑖𝑚 as thecontext of 𝑠𝑚 𝑗𝑘 .
As shown in Fig. 2, we modified the input vector for training asfollows:𝑣 (𝑡𝑚 𝑗𝑘𝑛 ) = 𝑣 onehot (𝑖𝑚 ), w2v(𝑡𝑚 𝑗𝑘 (𝑛−𝑤) ), · · · , w2v(𝑡𝑚 𝑗𝑘 (𝑛−1) ) ,(1)
where 𝑤 is the window size, w2v(𝑡) is a distributed expressionof term 𝑡 obtained using Word2Vec, and 𝑣 onehot (𝑖𝑚 ) is a one-hot
vector for item 𝑖𝑚 . The 𝑝-th dimension of 𝑣 onehot (𝑖𝑚 ) is defined asfollows:(
1 if 𝑝 = 𝑚,𝑣 onehot (𝑖𝑚 )𝑝 =(2)0 otherwise.
It uses a one-hot vector about the item instead of about the sentence.
Next, our method vectorizes all the sentences in the dataset using
the trained network. The vectorization procedures are identical asin the previous Doc2Vec. The middle layer’s values are used as thevector of a sentence in the vectorization phase. When used for asearch, a query is vectorized with the same model. To vectorize
a particular short text, it uses input vector 𝑣 (𝑡𝑚 𝑗𝑘𝑛 ) that contains
𝑚-dimensional zero vector instead of a one-hot vector. The vectorizations of the sentences in the dataset and the query are then used
for similarity calculation (e.g., cosine similarity).

3.3

Kurihara et al.

Similarity Calculation Using Query
Expansion

The vectorization of the keyword query and each sentence in thereview texts allows for similarity calculation. Then the method
can rank the evaluation expressions corresponding to the query.
Cosine similarity is one of the most general methods to compare
vectors. However, queries and sentences have a different volume
of information. A keyword query consisting of only one or twowords has a smaller range of expressions and fewer types than anevaluation expression of multiple words. Therefore, we increased

Arbitrary dimensional middle layer

One-hot vectorof Item imDistributed representation
vector of the term tmjk(n-2)

Distributed representation
vector of the term tmjk(n-1)

Figure 2: Input and output of training with window size 2.
The color depth of each cell reflects the value of each dimension.

the information content of the keyword query using the traditional
query expansion method before comparing it with each sentence.
When a given query was 𝑞, the method enumerates its lexicalsynonyms. Then it concatenates the synonyms into one short sentence form with Word2Vec by adding the top five synonyms to theoriginal query. For instance, when 𝑞 is a single term, “sad,” sentence𝑞 ′ becomes: “sad tragic sadness sorrow sorrowful regretful ...” In
this case, any corpus can be used to calculate the synonyms to theWord2Vec calculation, and another thesaurus can also be used forthis purpose. For our experiment, we just used the movie dataset
itself.
This expansion provides two benefits. First, if a query consistingof words is too narrowly defined, many results related to one specific word will appear at the top of the search results. By extending
the words at the lexical and conceptual levels, we can gather morediverse expressions. Second, the length of the sentence itself affects
the Doc2Vec calculation. If we vectorize sentences and words as
they are, there are only a few types of vectors that a word can take.
To improve the recall rate at the expense of the match rate, thequery should be extended in advance.
After extending the query, we vectorize the extended sentencesusing the above model. Then the method calculates the cosine similarity with all the sentences in the vectorized review and outputs
the results from the top.

4

EVALUATION

We experimentally verified the usefulness of the proposed method
using movie reviews, which are typical applications that can benefit.
We compared our new method against three baseline methods. Since
our final goal is to make a comfortable item search system based onreviewer opinions, we evaluated the methods with the metrics andmeasurements used in information retrieval research. We retrievedsentences from actual movie reviews for ten queries prepared inadvance whose meanings resemble the queries. After ranking thesentences, a crowdsourcing questionnaire evaluated the degreeof matching between the query and a selection of sentences. We
also qualitatively evaluated the variability of the expressions in theactual search results.

Doc2Vec-based Approach for Extracting Diverse
Evaluation Expressions from Online Review Data

4.1

DatasetWe used entire movie reviews posted on Yahoo! Movies, one ofJapan’s biggest online movie review sites. Yahoo! Movies provides
information on about 63,000 movies and more than 5 million reviews for those movies. We reduced the data size by selecting only
movies with 300 or more reviews, resulting in 3,000 movies, approximately 1.3 million reviews, and approximately 12 million sentences.
We preprocessed the data to make them suitable for Doc2Vec.
All the sentences were separated into words using a morphologicalanalyzer called MeCab. This step is important because Japanese
sentences are written without spaces between words.
Since it was impossible to calculate plain-D2V and LSI on thewhole dataset because of high memory requirements, we created asampled dataset by reducing 90% of the data. Each sentence in the
300 movie reviews was ranked by calculating the relevancy to thegiven query.

4.2

Evaluated Methods

Six methods were prepared for evaluation. Their details are as
follows:• TTA-D2V + QE: Target-Topic Aware Doc2Vec with Query
Expansion is the proposed method described in Section 3.
It evaluates the effectiveness of both the granularity manipulation of the training data in Doc2vec and the query
expansion.
• TTA-D2V: Target-Topic-Aware Doc2Vec is a variant method
to evaluate the effects caused by the target-topic alone.
• D2V + QE: Doc2Vec [13] with query expansion is anothervariant method to evaluate the effectiveness of query expansion alone. It combines query expansion and the original
Doc2Vec.
• D2V: Doc2Vec is a baseline method using the original Doc2Vec
without any changes.
• LSI: Latent Semantic Indexing [5] is another baseline methodthat uses topic modeling. We selected it because LSI is considered more suitable for short sentences than probabilistic
topic models (e.g., pLSA and LDA).
• random: Random Extracting ranks sentences randomly fromthe dataset for any query.
As a Doc2Vec implementation, we used gensim1 for the proposed
methods and D2V. The vector size is 200, and the window size is 7
in the Doc2Vec and TTA-Doc2Vec calculations. All other learningparameters are the default values of gensim. The number of topics(vector size) for LSI is 200, equal to the methods using Doc2Vec.

4.3

Queries

For the evaluation experiment, we derived ten queries from tags onmovie review sites, the categories of movie information sites, andfeature articles about movies. Table 1 shows the selected queries
and their features. These queries were used with each method, andreview sentences with high similarity were retrieved.

1 https://radimrehurek.com/gensim/

Conference’17, July 2017, Washington, DC, USA

4.4

Relevance Labeling with Crowdsourcing

We used a well-known crowdsourcing service for labeling the search
results. The participants scored the similarity between the shownqueries and the sentences on a scale from 1 to 4: completely different,
slightly different, slightly similar, and identical.
The number of questions was 100 for each of ten queries andfour methods, resulting in 4,000 questions. Because the method
aims to find expressions that are different from the query, sentencesthat contain the query term itself were removed from the search
results.
Sentences used for the questions were sampled from the 500
top-ranked results for each of the four methods. The sampling rate
was set high in the top part of the ranking and lower further down.
In addition to all of the top 30 sentences, 30 sentences from ranks
31 to 100 and 40 sentences from ranks 101 to 500 were randomly
selected.
Dummy questions were interspersed in the questionnaire to
weed out dishonest workers. They were occasionally asked to perform simple arithmetic calculations, such as whether a certain
number was even. Dishonest workers were removed, and the samequestionnaire was reassigned to another worker.

5

RESULTSThis section describes the experimental results from the viewpoints
of precision, ranking, and expression diversity. We collected 24,000
answers; four answers each prepared 4,000 questions.
To check the validity of the crowdsourcing results, we calculated the degree of agreement among the raters. Since the numberof subjects per task was four, we used Fleiss’s Kappa coefficient.
Since 𝜅 = 0.28 was between 0.21 to 0.40, these opinions have fairagreement [12].
We next compared the precision of the rankings retrieved byeach method. A sentence with an average score of 2.75 points ormore for the four answers is defined as relevant. Table 2 shows theprecision at 10, at 100, and at 500 of each method. A precision of
0.07 for the random extraction shows the difficulty of this searchtask. The dataset contains only seven % correct answers.
TTA-W2V performed the most accurate among the proposed
methods. The methods using the idea of target-topic and LSI achieved
higher precision than the others. The TTA-D2V method showedsignificantly higher precision than LSI (𝑝 = 0.00 on Welch’s 𝑡 test).
In contrast, the accuracy of method TTA-D2V + QE, which included all the extensions, did not differ from LSI. Fig. 3 shows theprecision in each section of the rankings. For methods other thanrandom, the precision steadily declined. The TTA-D2V method
had the highest precision in the top ranks and among the totals.
The ranking accuracy of each method was evaluated using normalized Discounted Cumulative Gain (nDCG). As Table 2 shows, theproposed method has a higher nDCG score than the other methods.
The expression diversity of the sentences in the results is anotherimportant factor. As an example, Table 3 shows the top five search
results of our new method and LSI for the query “Tear-jerker.”
The precision of both methods is almost the same for this query,although our method found more diverse expressions.

Conference’17, July 2017, Washington, DC, USA

Kurihara et al.

Table 1: Queries for movie review search task and their nDCG scores (translated)
Query

TTA-D2V+QE

TTA-D2V

D2V+QE

D2V

LSI

randomSurrealistic
Surprise ending
Familial love
Near-futuristic

0.85
0.58
0.59
0.70

0.73
0.86
0.89
0.74

0.70
0.62
0.69
0.69

0.70
0.62
0.55
0.69

0.69
0.74
0.61
0.63

0.64
0.53
0.49
0.49

Movie content

Relaxing
Nostalgic
Tear-jerker
Makes me want to go on a trip

0.72
0.79
0.71
0.69

0.75
0.71
0.88
0.78

0.71
0.65
0.76
0.52

0.74
0.69
0.66
0.53

0.63
0.73
0.78
0.72

0.49
0.48
0.46
0.45

Viewer sentiment

Suitable for a dateRewatchable

0.75
0.54

0.80
0.93

0.65
0.67

0.64
0.70

0.56
0.63

0.57
0.56

Situation

average

0.69

0.81

0.67

0.65

0.67

0.51

Table 2: Precision and nDCG (average of ten queries)

TTA-D2V+QE
TTA-D2V
D2V+QE
D2V
LSI
random

p@10

p@100

p@500

nDCG

0.44
0.72
0.32
0.32
0.46
0.11

0.56
0.36
0.34
0.31
0.31
0.36

0.29
0.51
0.32
0.29
0.32
0.07

0.69
0.81
0.67
0.65
0.67
0.51

Table 3: Top five results for proposed methods (TTA-D2V
and TTa-D2V+QE) and LSI for query “Tear-jerker” (translated).
MethodTTA-D2V + QE

Rank
1
2
3
4
5

0.8

Precision

2.25
3.00
3.75
1.25
4.00
4.00
3.50
3.75
3.00

LSI

1

Let’s all just cry together.
I cried, laughed,and was impressed.
Crying!
I cried.
I cried during the song.

3.25

2
3
4
5

0.1

4.00

The last scene made me cry. I sobbed!
This is a movie that make me cry.
I can’t help but cry now.
I cried!
Tearful!

0.5

0.2

Relevance

1
2
3
4
5

0.6

0.3

Sentence
I couldn’t stop mytears during the last scene.
It was a heart-warming movie.
Although I didn’t want to cry, I
couldn’t help myself during the last scene.
I was deeply movedby the leading actor.
I was scaredby that impression.

TTA-D2V
0.7

0.4

Type

3.00
2.00
4.00
3.50

0
1～10

11～20

21～30

TTA-D2V+QE

31～53
TTA-D2V

54～76 77～100 101～200201～300301～400401～500
D2V

D2V+QE

LSI

randomRanking

Figure 3: Precision for each method and ranking section

6

DISCUSSION

Our experimental results show that the TTA-D2V method is significantly more accurate than the others. Our method may be more
precise because it uses target-topics as contexts. Since method
TTA-D2V + QE using target-topics also outperformed the plain
Doc2Vec, considering the target-topic seems effective to vectorize
short sentences.
When query expansion was applied, the precision and nDCG
tended to decrease, indicating that acquiring more diverse representations provides a trade-off with accuracy. Although the accuracy

fell, the methods using query expansion extracted some evaluation
expressions that could not be obtained with the conventional methods. As Table 3 shows, the query expansion excavated such diverse
expressions as focusing on tears, handkerchiefs, and hearts for thequery “Tear-jerker.” Much onomatopoeic language (e.g., boohoo,phew) was included in the results of methods using query expansion. In contrast, LSI found many simple synonyms for queries.
Table 1 shows the nDCG for each query. TTA-D2V earned thehighest scores throughout, but for some queries, the query expansion increased their accuracy. For the queries related to viewer content, both our methods and LSI were highly accurate. In these tasks,
the viewer impressions are simply included in the review. Therefore,
when the method extracts ordinary synonyms, the crowdworkers
judged them to be relevant to the query. However, concerning
the actually obtained expressions, the proposed method extracted
various sentences that were not just synonyms.

Doc2Vec-based Approach for Extracting Diverse
Evaluation Expressions from Online Review Data

For another queried related to the movie content, our methods
outperformed the other existing methods. In these search tasks,
the methods had to extract specific descriptions or scenes fromthe movies. TTA-D2V obtained many relevant expressions. On
the other hand, the method’s accuracy was reduced using query
expansion. Since the range of expressions became too divergent inthese methods, too many irrelevant expressions were included in
the results.
Query expansion was particularly effective when the queries
were “surrealistic” and “makes me nostalgic.” These ambiguous
queries tended to be highly diverse expressions. The perceptionof the surreal is highly subjective. The query “nostalgic” was ambiguous because the query itself has multiple meanings. Youngpeople feel nostalgic about movies they saw in their childhood, andolder people long for older movies. Even a new movie might seemnostalgic if it is set in a previous era. A similar trend was foundin the query “surrealistic.” What people label as surrealistic differsfrom person to person. Query expansion increased the accuracy ofsuch ambiguous queries by giving them various meanings.

7

CONCLUSION

This paper proposed a Doc2Vec-based method to find actual expressions related to a given keyword query. This research contains two
ideas: focusing on target-topics and using query expansion before
similarity calculation. Through a large-scale evaluation with actual
movie review data, our method produced better ranking quality
and more diverse results.
Although the proposed method can obtain many evaluative expressions, it does not sufficiently consider the depth of impressions.
A practical application is also essential. At the moment, this method
is limited to matching the expressions in reviews with queries. In
practice, movies must be ranked from queries based on the obtained
expressions. To achieve this goal, we need to aggregate the review
information for each movie, summarize multiple reviews of different degrees, and sort them by their relevance to the query. We
also have to determine how to correctly aggregate a movie thatten people described as “sad” and a movie that one person labeled“very sad.” Instead of simple cosine similarity, we need to combine
our method with better ranking algorithms, such as deep learningor learning to rank.

ACKNOWLEDGMENTS
This work was supported by JSPS KAKENHI Grants Number 18K18161,
21H03775, and 18H03243.

REFERENCES[1] Nadeem Bader, Osnat Mokryn, and Joel Lanir. 2017. Exploring Emotions in Online
Movie Reviews for Online Browsing. In Proceedings of the 22Nd International
Conference on Intelligent User Interfaces Companion (Limassol, Cyprus) (IUI ’17
Companion). ACM, New York, NY, USA, 35–38. https://doi.org/10.1145/3030024.
3040982[2] Oren Barkan and Noam Koenigstein. 2016. Item2vec: neural item embedding
for collaborative filtering. In 2016 IEEE 26th International Workshop on Machine
Learning for Signal Processing (MLSP). IEEE, 1–6.
[3] Abd Samad Hasan Basari, Burairah Hussin, I Gede Pramudya Ananta, and Junta
Zeniarja. 2013. Opinion mining of movie review using hybrid method of support
vector machine and particle swarm optimization. Procedia Engineering 53 (2013),
453–462.

Conference’17, July 2017, Washington, DC, USA[4] Arti Buche, M. B. Chandak, and Akshay Zadgaonkar. 39 – 48. Opinion Mining
and Analysis: A survey. International Journal on Natural Language Computing 2,
3 (39 – 48).
[5] Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, andRichard Harshman. 1990. Indexing by latent semantic analysis. Journal of theAmerican society for information science 41, 6 (1990), 391–407.
[6] Fatemeh Hemmatian and Mohammad Karim Sohrabi. 2019. A survey on classification techniques for opinion mining and sentiment analysis. Artificial Intelligence
Review 52, 3 (2019), 1495–1545.
[7] Nan Hu, Paul A. Pavlou, and Jennifer Zhang. 2006. Can Online Reviews Reveal a
Product’s True Quality?: Empirical Findings and Analytical Modeling of Online
Word-of-mouth Communication. In Proceedings of the 7th ACM Conference on
Electronic Commerce (Ann Arbor, Michigan, USA) (EC ’06). ACM, New York, NY,
USA, 324–330. https://doi.org/10.1145/1134707.1134743[8] Yohan Jo and Alice H. Oh. 2011. Aspect and Sentiment Unification Model for
Online Review Analysis. In Proceedings of the Fourth ACM International Conferenceon Web Search and Data Mining (Hong Kong, China) (WSDM ’11). ACM, New
York, NY, USA, 815–824. https://doi.org/10.1145/1935826.1935932[9] Kenji Sugiki and Shigeki Matsubara. 2007. A product retrieval system robust to
subjective queries. In 2007 2nd International Conference on Digital Information
Management, Vol. 1. 351–356. https://doi.org/10.1109/ICDIM.2007.4444248[10] Jayashri Khairnar and Mayura Kinikar. 2013. Machine learning algorithms for
opinion mining and sentiment classification. International Journal of Scientificand Research Publications 3, 6 (2013), 1–6.
[11] Kosuke Kurihara, Yoshiyuki Shoji, Sumio Fujita, and Martin J. Dürst. 2019. TargetTopic Aware Doc2Vec for Short Sentence Retrieval from User Generated Content.
In Proceedings of the 21st International Conference on Information Integration and
Web-Based Applications & Services (Munich, Germany) (iiWAS2019). Association
for Computing Machinery, New York, NY, USA, 463–467.
[12] J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer
Agreement for Categorical Data. Biometrics 33, 1 (1977), 159–174. http://www.jstor.org/stable/2529310[13] Quoc Le and Tomas Mikolov. 2014. Distributed Representations of Sentences andDocuments. In Proceedings of the 31st International Conference on International
Conference on Machine Learning - Volume 32 (Beijing, China) (ICML’14). JMLR.org,
II–1188–II–1196. http://dl.acm.org/citation.cfm?id=3044805.3045025[14] Bing Liu and Lei Zhang. 2012. A survey of opinion mining and sentiment analysis.
In Mining text data. Springer, 415–463.
[15] Chien-Liang Liu, Wen-Hoar Hsaio, Chia-Hoang Lee, Gen-Chi Lu, and Emery
Jou. 2011. Movie rating and review summarization in mobile environment. IEEE
Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)
42, 3 (2011), 397–407.
[16] Gaojun Liu and Xingyu Wu. 2019. Using collaborative filtering algorithms
combined with Doc2Vec for movie recommendation. In 2019 IEEE 3rd Information
Technology, Networking, Electronic and Automation Control Conference (ITNEC).
IEEE, 1461–1464.
[17] Van-Thuy Phi, Liu Chen, and Yu Hirate. 2016. Distributed representation based
recommender systems in e-commerce. In DEIM Forum.
[18] Vijay B Raut and DD Londhe. 2014. Opinion mining and summarization of
hotel reviews. In 2014 International Conference on Computational Intelligence andCommunication Networks. IEEE, 556–559.
[19] Sumbal Riaz, Mehvish Fatima, Muhammad Kamran, and M Wasif Nisar. 2019.
Opinion mining on large scale data using sentiment analysis and k-means clustering. Cluster Computing 22, 3 (2019), 7149–7164.
[20] Vivek Kumar Singh, Rajesh Piryani, Ashraf Uddin, and Pranav Waila. 2013. Sentiment analysis of movie reviews: A new feature-based heuristic for aspect-level
sentiment classification. In 2013 International Mutli-Conference on Automation,
Computing, Communication, Control and Compressed Sensing (iMac4s). 712–717.
https://doi.org/10.1109/iMac4s.2013.6526500[21] Jiaxing Tan, Alexander Kotov, Rojiar Pir Mohammadiani, and Yumei Huo. 2017.
Sentence Retrieval with Sentiment-specific Topical Anchoring for Review Summarization. In Proceedings of the 2017 ACM on Conference on Information andKnowledge Management (Singapore, Singapore) (CIKM ’17). ACM, New York, NY,
USA, 2323–2326. https://doi.org/10.1145/3132847.3133153[22] Ivan Titov and Ryan McDonald. 2008. A joint model of text and aspect ratings
for sentiment summarization. In proceedings of ACL-08: HLT. 308–316.
[23] Lap Q. Trieu, Huy Q. Tran, and Minh-Triet Tran. 2017. News Classificationfrom Social Media Using Twitter-based Doc2Vec Model and Automatic Query
Expansion. In Proceedings of the Eighth International Symposium on Information
and Communication Technology (Nha Trang City, Viet Nam) (SoICT 2017). ACM,
New York, NY, USA, 460–467. https://doi.org/10.1145/3155133.3155206[24] Christophe Van Gysel, Maarten de Rijke, and Evangelos Kanoulas. 2018. Mix ’N
Match: Integrating Text Matching and Product Substitutability Within ProductSearch. In Proceedings of the 27th ACM International Conference on Information
and Knowledge Management (Torino, Italy) (CIKM ’18). ACM, New York, NY, USA,1373–1382. https://doi.org/10.1145/3269206.3271668[25] Libing Wu, Cong Quan, Chenliang Li, and Donghong Ji. 2018. PARL: Let Strangers
Speak Out What You Like. In Proceedings of the 27th ACM International ConferenceConference’17, July 2017, Washington, DC, USA

on Information and Knowledge Management (Torino, Italy) (CIKM ’18). ACM, New
York, NY, USA, 677–686. https://doi.org/10.1145/3269206.3271695[26] Lili Zhao and Chunping Li. 2009. Ontology based opinion mining for movie
reviews. In International Conference on Knowledge Science, Engineering and Management. Springer, 204–214.
[27] Li Zhuang, Feng Jing, and Xiao-Yan Zhu. 2006. Movie Review Mining and Summarization. In Proceedings of the 15th ACM International Conference on Information

Kurihara et al.

and Knowledge Management (Arlington, Virginia, USA) (CIKM ’06). ACM, New
York, NY, USA, 43–50. https://doi.org/10.1145/1183614.1183625[28] Yuan Zuo, Junjie Wu, Hui Zhang, Hao Lin, Fei Wang, Ke Xu, and Hui Xiong. 2016.
Topic Modeling of Short Texts: A Pseudo-Document View. In Proceedings of the
22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data
Mining (San Francisco, California, USA) (KDD ’16). ACM, New York, NY, USA,2105–2114. https://doi.org/10.1145/2939672.2939880