Target-Topic Aware Doc2Vec for
Short Sentence Retrieval from User Generated Content
Kosuke Kurihara

Yoshiyuki Shoji

Aoyama Gakuin University
Kanagawa, Japankurihara@sw.it.aoyama.ac.jp

Aoyama Gakuin University
Kanagawa, Japanshoji@it.aoyama.ac.jp

Sumio Fujita

Martin J. Dürst

Yahoo JAPAN Corporation
Tokyo, Japansufujita@yahoo-corp.jp

Aoyama Gakuin University
Kanagawa, Japanduerst@it.aoyama.ac.jp

ABSTRACTThis paper proposes a new method of supplementing the context
of short sentences for the training phase of Doc2Vec. Since CGM(Consumer Generated Media) sites and SNS sites become widespread, the importance of similarity calculation between a given
query and a short sentence is increasing. As an example, a searchby the query “sad” should find actual expressions such as “I neededa handkerchief” on a movie review site. Doc2Vec is one of the mostwidely used methods for vectorization of queries and sentences.
However, Doc2Vec often exhibits low accuracy if the training dataconsists of short sentences, because they lack context. We modified
Doc2Vec with the hypothesis that other posts for the same topic (i.e.
reviews for the same movie in online movie review sites) may sharethe same background. Our method uses target-topic IDs instead ofsentence IDs as the context in the training phase of the Doc2Vec
with the PV-DM model; this model estimates the next term from a
few previous terms and context. The model trained with item IDs
vectorizes a sentence more accurately than a model trained withsentence IDs. We conducted a large-scale experiment using 1.2 million movie review posts and a crowdsourcing-based evaluation. The
experimental result demonstrates that our new method achieves
higher precision and nDCG than previous Doc2Vec variants andtraditional topic modeling methods.

query. However, Doc2Vec exhibits low accuracy in learning short
sentences because short sentences do not provide enough context.
This paper proposes a method to supplement for the lack of context
during the training phase of Doc2Vec when learning short sentencesfrom the social sites. Posts in such kinds of sites generally have atarget-topic. For instance, a social media post often contains one ormore hashtags, a review in an online review site has a target item,
and a comment in an online discussion forum has a news article as
the topic. Here we introduce the term “target-topic” for the objects
or referents of these sentences. We modify Doc2Vec by using targettopics as an additional context for training the Doc2Vec network.
The expectation is that this allows to exploit the similarity between
sentences that relate to the same target-topic, and will therefore
improve search accuracy.
A typical application is movie search using reviews: Imagine youwant to watch a movie that makes you feel nostalgic. Using thesearch term “nostalgic”, it should be possible to find reviews withcontain text fragments such as “brings back memories” or “reminds
me of my childhood”. Using vectorization including target-topics,
we significantly improve the accuracy of the similarity calculation
between queries and target sentences.
Details of the new method are given in Section 3. Section 4
describes the experimental setup and Section 5 provides the results
to show the effectiveness of using target-topics.

KEYWORDS
Online Review Sites, Doc2Vec, Information Retrieval, Impression
Retrieval

1

INTRODUCTION

The present-day Web can be seen both as a collection of fragmented
sentences and as a set of documents. Due to the rise of CGM (Consumer Generated Media) sites and SNS (Social Networking Service)sites, many people frequently post a large number of short sentenceson the Web. The Web contains many by-products of communication and short posts of self-expression. Following such change ofproperty of the Web resources, the importance of similarity calculation between short sentences and queries is increasing. Most of
the existing search algorithms mainly target larger documents, butthere are few suitable algorithms for searching Web resources witha lot of fragmented sentences.
Doc2Vec [6] is one of the most popular methods to vectorize
sentences and to calculate the similarity between sentences and a

2

RELATED WORK

We introduce related work from the viewpoint of our proposed
method and our application.
Our method is an instance of information retrieval with distributed expressions. There are many methods using Word2Vec or
Doc2Vec to find information from social sites. Van Gysel et al. [10]also used Doc2Vec models for short sentences in the social sites. It is
active research field that tackle short sentences in social sites, Trieu
et al. [9] propose a method for tagging and classifying news information posted on Twitter and searching for similar news. Neitherof these use target-topics to improve search accuracy. Zuo [13] et.al
uses the external information to vectorize short sentences in socialsites for probabilistic topic models. Our method also uses external
context for Doc2Vec-based vectorization. The main difference isthat we use target-topic as an external information; our method
does not need any ontologies or dictionaries.

Kurihara, et al.

iiWAS2019, pre-print, author version
Online Review Site (I, R, S, T)
Reviews
Review r11

Sentence s111

s112
t1121 t1122

Term t1111

t1112

s121
t1211 t1212

Review r12
s122
t1221

Distributed representation vector of the term tmjkn

Items(Target Topic)
Arbitrary dimensional middle layer

Item i1

Review rmj

Sentence smjk
Term tmjkn

Figure 1: Site structure of a typical online review site. Everysentence points to one item (target-topic).
Online review analysis is the second research field strongly related to our proposal. Online reviews are a powerful information
resource for item retrieval [5], recommendation [11], decision support [3, 12], and so on. Singh et al. [7] and Bader et al. [1] focus
on expressions and sentiments in reviews. Jo et al. [4] propose amethod that automatically detects the combination of various aspects and polarities in reviews. Tan et al. [8] propose another way
to find short sentences which have the similar sentiment. There
are two advantages of our method when compared to previous
research. It can search for sentences with any aspects, not only
sentiments but also story patterns or genres. Also, it does not needpreparing a sentiment label dictionary in advance.

3One-hot vectorof Item im

Item imTARGET-TOPIC AWARE DOC2VEC

Overall, our proposed method is a variant of the PV-DM (Paragraph
Vector Distributed Memory) model of Doc2Vec. First, a two-layer
network is trained through the task of estimating the next termin a sentence from its context. In the original PV-DM model, aseparate context is used for each sentence. We modify this by using
the target-topic as the context for all sentences about this targettopic. Second, each sentence is vectorized using the trained network.
This step is the same as in the original model, but as a result, thegranularity is different for the training step and the vectorization
step.
To explain the details of our new method, we use an online
review site as an example; it is one of the most typical examples ofa CGM site. Online review sites have a common structure as shownin Figure 1. In this example, we call target-topic of a review as “item”
for clarity. A review site consists of items (target-topics) I , reviews
R, sentences S and terms T . Each item im is discussed by a numberof reviews. Each review rmj for item im consists of a number ofsentences. Each sentence smjk is a sequence of terms denoted bytmjkn . The goal of our method is to vectorize smjk accurately. For
that purpose, our method uses item im as the context of smjk .
As shown in Figure 2, we modified the input vector for training
as follows:
v(tmjkn ) = v onehot (im ), w2v(tmjk (n−w ) ), · · · , w2v(tmjk (n−1) )(1)

Distributed representation
vector of the term tmjk(n-2)

Distributed representation
vector of the term tmjk(n-1)

Figure 2: Input and output of training with window size 2.
The color depth of each cell reflects the value of each dimension.

where w is the window size, w2v(t) is a distributed expression ofterm t obtained by using Word2Vec, and v onehot (im ) is the one-hot
vector for item im . The p-th dimension of v onehot (im ) is defined asfollows:(
1 if p = m,v onehot (im )p =(2)0 otherwise.
It uses one-hot vector about the item instead of about the sentence.
Next, our method vectorizes all the sentences in the dataset byusing the trained network. The vectorization procedures are thesame to the previous Doc2Vec. The values of the middle layer areused as the vector of a sentence in the vectorization phase. When it
was used for the search, a query is vectorized with the same trainedmodel. To vectorize a certain short text, it uses the input vectorv(tmjkn ) that contains an m-dimensional zero vector instead of
one-hot vector. The vectorizations of the sentences in the dataset
and the query are then used for similarity calculation (e.g. cosine
similarity).

4

EXPERIMENT

We verified the usefulness of the proposed method by an experiment using movie reviews. Movie reviews are a typical application
which can benefit from the proposed method. We compared our
new method against three baseline methods. Since our final goal
is to make a comfortable item search system based on reviewers’
opinions, we evaluated methods with the metrics and the measurement used in information retrieval research. For 10 queries
prepared in advance, we retrieved actual movie review sentencesthat have similar meanings to the queries. After ranking the sentences, a crowdsourcing questionnaire was used to evaluate thedegree of matching between the query and a selection of sentences.
In addition, we qualitatively evaluated the variability of expression
in the actual search results.

4.1

DatasetWe used actual movie review data posted on Yahoo! Movies, oneof the biggest online movie review site in Japan. We created movie
review dataset which consist of 3,245 movies. Each movie in theTarget-Topic Aware Doc2Vec for
Short Sentence Retrieval from User Generated Content
dataset has 300 or more reviews. The dataset contains approximately 1.3 million reviews, and approximately 12 million sentencesin total.
We preprocessed the data to make it suitable for Doc2Vec. All
sentences were separated into words using a Japanese morphological analyzer. This is an important step because Japanese text iswritten without spaces between words. Word classes were limited
to nouns, adjectives, and verbs. This is different from most otherresearch, where only nouns are used. The reason for this differenceis that information such as emotions and impressions at the timeof watching a movie, and the situation suitable for watching themovie are also important.

4.2

iiWAS2019, pre-print, author version
Table 1: Queries used for movie review search task
(translated from Japanese)Querysurrealistic
surprise ending
familial lovenear-futuristic
makes me relaxed
makes me sadmakes me nostalgic
makes me want to
go on a trip
suitable for a daterewatchable

Evaluated Methods

Four methods were prepared for evaluation. The details of the four
evaluated methods are as follows:• TTA-D2V: Target-Topic-Aware Doc2Vec is the proposed
method described in Section 3.
• plain-D2V: Plain Doc2Vec [6] is a baseline method using
Doc2Vec without any changes.
• LSI: Latent Semantic Indexing [2] is a baseline method using
topic modeling. We selected this method because LSI is considered more suitable for short sentences than probabilistic
topic models (e.g. pLSA and LDA).
• random: Random Extracting ranks sentences randomly fromthe dataset for any query.
As a Doc2Vec implementation, we used gensim1 for the proposed
method and plain-D2V. The vector size is 200 and the window
size is 7. All other learning parameters are the default values ofgensim. The number of topics (vector size) for LSI is 200, equal tothe methods using Doc2Vec.
Note that, because of high memory requirements, it was notpossible to calculate plain-D2V and LSI on the whole dataset. For
these two methods, the data was randomly reduced to a 10% subset.

4.3

Queries

For the evaluation experiment, we selected 10 queries. The selected
queries were derived from tags on movie review sites, categories ofmovie information sites, and feature articles about movies. Table 1
shows the selected queries and their features. Each query can beroughly classified into three types: representing the scene drawn by
the movie, representing emotions and impressions that people havewhen watching the movie, and representing a situation suitable forwatching the movie. These queries were used with each method,
and a review sentences with high similarity were retrieved.

4.4

Relevance Labeling with Crowdsourcing

We used Yahoo! Japam Crowdsourcing, that is a well-known Japanese crowdsourcing service, for labeling the search results. The participants were asked to score the similarity between the shown
query and the sentence on a scale from 1 to 4: completely different,
slight different, slight similar, and completely similar.
The number of questions was 100 for each of 10 queries and for
four methods, resulting in a total of 4,000 questions. Because the1 https://radimrehurek.com/gensim/

Type
movie
contents

viewersentiment

situation

aim of the method is to find expressions different from the query,sentences which contain the query term itself were removed fromthe search results.
Sentences used for the questions were sampled from the 500
top-ranked results for each of the four methods. The sampling rate
was set high in the top part of the ranking, and lower further down.
In addition to all of the top 30 sentences, 30 sentences from rank
31 to rank 100 and 40 sentences from rank 101 to rank 500 wererandomly selected.

5

RESULTSThis section describes the experimental results from the view points
of precision, ranking, and expression diversity. In the experiment,
16,000 answers were collected from 298 crowd workers.
The precision of rankings retrieved by each method is compared.
A sentence with an average score of 2.75 points or more for thefour answers is defined as relevant. Table 2 shows the precision at
100 and precision at 500 of each method. A precision of 0.07 forrandom extraction means that the dataset contains around sevenpercent of correct answers. The method proposed and LSI archived
higher precision than the other methods. The new method shows
significantly higher precision than LSI (p = 0.00 on Welch’s t test).
Figure 3 shows the precision in each section of the rankings. The
proposed method has the highest precision both in the top ranks
and in total.
The ranking accuracy of each method was evaluated using nDCG
(normalized Discounted Cumulative Gain). As table 2 shows, theproposed method has a higher nDCG score than the other methods.
Expression diversity of the sentences in the results is also an
important factor. As an example, Table 3 shows the top five search
results of our new method and of LSI for the query “makes me sad”.
The precision of both methods is almost the same for this query,but our method found more diverse expressions.

6

DISCUSSION

The experimental results show that our method is significantly moreaccurate than other methods. This can be attributed to one of two
reasons, or a mixture of both. First, our new method may be more
precise because it uses target-topics as contexts. Second, LSI andKurihara, et al.

iiWAS2019, pre-print, author version
0.6

Precision

0.5
0.4
0.3
0.2
0.1
0

Ranking

TTA-D2V

plain-D2V

LSI

randomFigure 3: Precision for each method and ranking section
Table 2: Precision and nDCG (Average of 10 queries)

TTA-D2V
plain-D2V
LSI
random

p@100
0.45
0.31
0.36
0.08

p@500
0.43
0.29
0.32
0.07

nDCG
0.74
0.65
0.67
0.51

Table 3: Top 5 results for proposed method (TTA-D2V) and
LSI for the query “makes me sad” (translated from Japanese).
MethodTTA-D2V

Rank
1
2
3
4
5

LSI

1
2
3
4
5

Sentence
Sentimental people
and those who like dogs
need a handkerchief.
This moved me to tears.
I sobbed.
You need a handkerchief.
People are in tears,
but I can’t understand.
Let’s all just cry.
I cried, laughed,and was impressed.
Cry!!
I cried.
I cried during the song.

Relevance
3.75
3.25
3.75
3.50
2.25
3.25
3.00
2.00
4.00
3.50

plain-D2V may be less precise because they use a smaller dataset,
which results from an explosion of the number of dimensions. A
more detailed analysis is needed. In any case, our proposed method
is likely to be useful for the vectorization of short sentences withtarget-topics.
With respect to the type of query, Table 4 shows that LSI obtained higher precision for three queries. In the case of the queries“surprise ending” and “familial love”, LSI found a few effective synonyms for the query (i.e., “last scene” for “surprise ending”). Thisshows that LSI is useful for tasks which can be solved by usingsynonyms. For one query, “makes me want to go on a trip”, theprecision of the proposed method was not very high. Table 5 showsthe example of sentences in the top search results for the query“makes me want to go on a trip”. For queries which consists of asequence of general terms, Doc2Vec and its variants tend to produce
a vague vector. Further improvements may be possible by selecting
the method based on the type of query.

7

CONCLUSION

This paper proposed a Doc2Vec-based method to vectorize short
sentences included in social sites. By focusing on the target-topic,
the proposed method can supplement the context of short sentences
for the training phase of Doc2Vec. Through a large-scale evaluation with actual movie review data, we showed that our method
produces better ranking quality and more diverse results.
A more detailed analysis is needed to understand the advantagesof our new method better. This includes comparing the methods byusing a dataset of the same size for all methods. The application is
also important. We plan to verify the usefulness of our method inother fields. We will also seek advanced ways to use the vectorized
sentences, such as embedding for deep learning.

ACKNOWLEDGMENTS
This work was supported by JSPS KAKENHI Grants Number 18K18161
and 18H03243.

REFERENCES[1] Nadeem Bader, Osnat Mokryn, and Joel Lanir. 2017. Exploring Emotions in Online
Movie Reviews for Online Browsing. In Proceedings of the 22Nd International
Conference on Intelligent User Interfaces Companion (IUI ’17 Companion). ACM,
New York, NY, USA, 35–38. https://doi.org/10.1145/3030024.3040982[2] Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, andRichard Harshman. 1990. Indexing by latent semantic analysis. Journal of theAmerican society for information science 41, 6 (1990), 391–407.
[3] Nan Hu, Paul A. Pavlou, and Jennifer Zhang. 2006. Can Online Reviews Reveal a
Product’s True Quality?: Empirical Findings and Analytical Modeling of Online
Word-of-mouth Communication. In Proceedings of the 7th ACM Conference on
Electronic Commerce (EC ’06). ACM, New York, NY, USA, 324–330. https://doi.
org/10.1145/1134707.1134743[4] Yohan Jo and Alice H. Oh. 2011. Aspect and Sentiment Unification Model for
Online Review Analysis. In Proceedings of the Fourth ACM International Conferenceon Web Search and Data Mining (WSDM ’11). ACM, New York, NY, USA, 815–824.
https://doi.org/10.1145/1935826.1935932

Target-Topic Aware Doc2Vec for
Short Sentence Retrieval from User Generated Content

iiWAS2019, pre-print, author version

Table 4: nDCG and p@k for each method and individual queries

Querysurrealistic
surprise ending
familial lovenear-futuristic
makes me relaxed
makes me sadmakes me nostalgic
makes me want to go on a trip
suitable for a daterewatchable

TTA-D2V
nDCG p@500
0.74
0.28
0.71
0.28
0.64
0.25
0.70
0.38
0.81
0.69
0.83
0.78
0.74
0.48
0.77
0.30
0.77
0.36
0.76
0.50

Table 5: Top 5 results for proposed method (TTA-D2V) and
LSI for the query “makes me want to go on a trip” (translated
from Japanese).
MethodTTA-D2V

Rank
1
2
3
4
5

LSI

1
2
3
4
5

Sentence
I want to go there.
I watched this movie
while I was travelling.
I would better go together
with somebody.
It contains a trip scene.
This movie wasplayed in mytrip destination.
I want to go there
after a long time
I wanted to visit
Machu Picchu.
I wanted so muchto go fishing.
I wanted to go to
the airport.
I wanted to eat noodles,
and I eat.

Relevance
3.25
2.50
3.25
3.00
2.50
2.25
4.00
2.25
3.25
3.00

[5] Kenji Sugiki and Shigeki Matsubara. 2007. A product retrieval system robust to
subjective queries. In 2007 2nd International Conference on Digital Information
Management, Vol. 1. 351–356. https://doi.org/10.1109/ICDIM.2007.4444248[6] Quoc Le and Tomas Mikolov. 2014. Distributed Representations of Sentences andDocuments. In Proceedings of the 31st International Conference on International
Conference on Machine Learning - Volume 32 (ICML’14). JMLR.org, II–1188–II–
1196. http://dl.acm.org/citation.cfm?id=3044805.3045025[7] Vivek Kumar Singh, Rajesh Piryani, Ashraf Uddin, and Pranav Waila. 2013. Sentiment analysis of movie reviews: A new feature-based heuristic for aspect-level
sentiment classification. In 2013 International Mutli-Conference on Automation,
Computing, Communication, Control and Compressed Sensing (iMac4s). 712–717.
https://doi.org/10.1109/iMac4s.2013.6526500[8] Jiaxing Tan, Alexander Kotov, Rojiar Pir Mohammadiani, and Yumei Huo. 2017.
Sentence Retrieval with Sentiment-specific Topical Anchoring for Review Summarization. In Proceedings of the 2017 ACM on Conference on Information andKnowledge Management (CIKM ’17). ACM, New York, NY, USA, 2323–2326.
https://doi.org/10.1145/3132847.3133153[9] Lap Q. Trieu, Huy Q. Tran, and Minh-Triet Tran. 2017. News Classificationfrom Social Media Using Twitter-based Doc2Vec Model and Automatic Query
Expansion. In Proceedings of the Eighth International Symposium on Information

plain-D2V
nDCG p@500
0.70
0.17
0.62
0.19
0.55
0.10
0.69
0.37
0.74
0.51
0.66
0.52
0.69
0.43
0.70
0.06
0.64
0.17
0.53
0.39

nDCG
0.69
0.74
0.62
0.63
0.63
0.78
0.74
0.63
0.56
0.72

LSI
p@500
0.16
0.40
0.28
0.18
0.37
0.77
0.46
0.33
0.05
0.23

randomnDCG p@500
0.64
0.09
0.53
0.08
0.49
0.05
0.49
0.03
0.49
0.05
0.46
0.08
0.48
0.08
0.56
0.03
0.57
0.07
0.45
0.16

and Communication Technology (SoICT 2017). ACM, New York, NY, USA, 460–467.
https://doi.org/10.1145/3155133.3155206[10] Christophe Van Gysel, Maarten de Rijke, and Evangelos Kanoulas. 2018. Mix ’N
Match: Integrating Text Matching and Product Substitutability Within ProductSearch. In Proceedings of the 27th ACM International Conference on Information
and Knowledge Management (CIKM ’18). ACM, New York, NY, USA, 1373–1382.
https://doi.org/10.1145/3269206.3271668[11] Libing Wu, Cong Quan, Chenliang Li, and Donghong Ji. 2018. PARL: Let Strangers
Speak Out What You Like. In Proceedings of the 27th ACM International Conferenceon Information and Knowledge Management (CIKM ’18). ACM, New York, NY,
USA, 677–686. https://doi.org/10.1145/3269206.3271695[12] Li Zhuang, Feng Jing, and Xiao-Yan Zhu. 2006. Movie Review Mining andSummarization. In Proceedings of the 15th ACM International Conference on Information and Knowledge Management (CIKM ’06). ACM, New York, NY, USA,
43–50. https://doi.org/10.1145/1183614.1183625[13] Yuan Zuo, Junjie Wu, Hui Zhang, Hao Lin, Fei Wang, Ke Xu, and Hui Xiong.
2016. Topic Modeling of Short Texts: A Pseudo-Document View. In Proceedingsof the 22Nd ACM SIGKDD International Conference on Knowledge Discovery andData Mining (KDD ’16). ACM, New York, NY, USA, 2105–2114. https://doi.org/
10.1145/2939672.2939880