BERT-Based Movie Keyword Search Leveraging
User-Generated Movie Rankings and Reviews
Tensho Miyashita

Yoshiyuki Shoji

Sumio Fujita

Graduate School of Science and Engineering
Aoyama Gakuin University
Kanagawa, Japantensho@sw.it.aoyama.ac.jp

Faculty of Informatics
Shizuoka University
Shizuoka, Japanshojiy@inf.shizuoka.ac.jp

LY Corporation
Tokyo, Japansufujita@lycorp.co.jp

Martin J. Dürst
College of Science and Engineering
Aoyama Gakuin University)
Kanagawa, Japanduerst@it.aoyama.ac.jp

Abstract—This paper introduces a novel method for movie
keyword searches based on user-generated rankings and reviews.
We utilize the capabilities of the BERT language model, which
has been enriched with task-specific fine-tuning. The model istrained to understand the relationship between keywords andmovies using paired user-generated ranking titles and movie
reviews. We sourced our data from a renowned Japanese movie
review platform. This dataset comprises 10,000 user rankings and15,000 films. In a binary classification task, our model demonstrated superior performance compared to traditional similaritybased methods. While our approach outperforms traditional
similarity methods, further improvements in pooling techniques
are necessary.
Index Terms—LLM, Movie Search, User Generated Contents,
Review, Ranking

I. I NTRODUCTION
In recent years, how people select movies has changed dueto the widespread use of on-demand and subscription-based
movie distribution services. Conventionally, a person goes to amovie theater and selects a movie to watch on the spot. In thisway, since only a few movies are played in theaters, they only
have to choose one movie to watch from several candidates.
Even when using video rental stores, the number of videos islimited; therefore, users can decide on a movie to watch before
coming to the store or search for a movie to watch in store.
However, when using movie distribution sites, users need to
select a movie from almost all movies ever released. In thiscase, it is difficult to examine each movie individually to see
if it suits their demands.
Attitudes toward watching movies are also changing. Recently, more and more people have been casually watchingmovies on their computers and smartphones. Watching movies
while working and increasing playback speed is becoming
more common; watching movies is becoming more integrated
into everyday life. This change in movie-watching style has
increased the demand for technology that enables users tosearch for movies more easily.

For example, there is a need to search for movies using the
ranking of arbitrary keywords, such as “I want to see movies
with beautiful scenery that I can play while working” or “I justwant to see movies with flashy action.” However, conventional
movie searches cannot find movies using keywords such as“beautiful scenery” and “flashy action” because movie information sites target descriptions and metadata. Metadata on
movie information websites generally only include informationsuch as the original title, country of production, director,
music, actors, etc., as well as descriptions and synopses.
However, the description or synopsis of a movie is merely
a text that describes what the movie is about from a third
party’s point of view. It does not include information about
the impressions the movie made on people who watched it.
In this situation, reviews are an important source of information for judging how viewers feel about a movie, such aswhether the movie has “beautiful scenery.” However, whentrying to find movies with “beautiful scenery” from the vast
number of movies available on video distribution sites, it is
not realistic to read the reviews posted for each movie one by
one to determine whether it is a movie that the user wants tosee.
In this paper, we propose a method for ranking movies
based on free queries by learning the relevance of movies
and queries from user-generated rankings and movie reviews
using bidirectional encoder representations from transformers(BERT). For this purpose, we focused on services that allow
users to submit their own movie rankings. Some movie review
sites have a function in which individuals can create a list of
their favorite movies. For example, Yahoo! Movies, one of thebiggest movie information sites in Japan, uses Round-Up toprovide this function, where users can register up to 10 movies
using any point of view they like. Each user-generated ranking
in Yahoo! Movies’ Round-Up is given a title, for example,“Top 10 movies that make me cry,” and then 10 movies arelisted.

We hypothesized that these ranking titles might function likesearch keywords. Therefore, we attempted to make movies
searchable using keywords by learning the relationship between ranking titles and the movies listed in the ranking. In
recent years, inference using large-scale language models has
become more common, as is typified by BERT. If such alanguage model can infer that “movies with this type of review
tend to appear in rankings with this kind of title,” then movies
can be searched for using arbitrary keywords. In other words,
movies can be scored by the probability of their appearance
in user-generated rankings whose titles contain the keyword
query.
To achieve such a search method, we proposed an algorithm
comprising three steps (see Figure 1):• Vectorize ranking titles and all reviews using BERT,• train the neural network model to calculate the relevance
between the ranking title and reviews of a certain movie,
and• generate rankings from a user-entered query.
The first step was the vectorization of keywords and reviews. We used BERT, one of the most popular large language
models, to vectorize sentences consisting of natural language,
such as queries and reviews. The feature vector archived by
BERT is a numerical representation of the meaning of asentence or word.
The second step involved learning the relevance of thequeries and reviews using machine learning. We used a simple
neural network model and trained it with movie vectors andranked title vectors. The user-generated rankings were used
for this learning as the training data. For example, supposea certain movie appeared in a ranking named “My top 10
tearjerker movies.” The model would then learn the movie
review and the ranking title as a pair. Such learning enables themodel to solve binary classification tasks for pairs of keywords
and reviews. For example, whether a movie with a particularreview appears in a user-generated ranking that includes the
keyword in the title.
The third step was to generate actual rankings from userentered queries. Here, the keyword query was vectorized using
BERT. The algorithm combined the vector of queries with thevector of movies and then classified them using the trainedmodel. That is, whether the movie is likely to appear in
the ranking with the query in its title. The algorithm canrank movies according to their classification probability by
performing a brute-force classification of all candidate movies.
Here, we expected the performance of BERT’s zero-shot
inference to be high enough to rank movies even by keywords
that were not included in the ranking title.
In this way, generating a movie ranking based on arbitrary
keywords was possible. We conducted a subject experiment
to confirm the accuracy and effectiveness of this ranking.
Participants were asked to evaluate how closely the movies
in the rankings generated by the proposed method and several
comparison methods were related to the query.
This paper is an extended version of the short paper previously published [1].

II. R ELATED W ORKThis research aimed to make movies searchable using wordof mouth (eWOM). BERT and Learning to Rank were usedas enabling technologies.
A. Item Search Using eWOM
Our research used reviews posted on movie sites, a type ofeWOM, to find movies that were close to what users werelooking for. There are several examples of this kind of item
search that focus on eWOM.
Ramanand et al. [2] proposed a method for extracting“wishes” that indicated suggestions about products and services and purchase intentions from documents, such as reviews
and buyer surveys. The “wishes” extracted from the reviews
were applied not only to improve the quality of products andservices but also the recommendations and presentations of
products sought by customers. Similarly, this study used user
review information to search for movies using free queries.
B. Information Retrieval Using BERT
BERT is a natural language processing model proposed by
Devlin et al. [3] that enables context reading. BERT has been
applied in various fields because it can convert the meaning ofwords in a sentence into a distributed representation of vectors
with high accuracy.
Depending on the application, BERT has been used for various task-specific purposes by fine-tuning pretrained language
models. As an example of fine-tuning focusing on sentence
similarity, Nils et al. [4] proposed Sentence Bert, a BERTbased model specialized for sentence similarity searches. Experiments evaluating the performance of sentence embedding
using SentEval have shown that this method can vectorize themeanings of sentences more effectively.
As another example of a specific domain, Zhuang et al. [5]proposed a fine-tuning BERT model that focuses on words
related to finance, FinBERT. Using performance evaluation
experiments using FiQA, a dataset consisting of questionand-answer documents in the field of finance showed that
FinBERT effectively reflects the meanings of finance-related
words in vectors. Shibata et al. [6] proposed a method
for retrieving query-relevant FAQs using BERT. Experiments
using the localgovFAQ and StackExchange datasets showedthat this method enabled more accurate FAQ retrieval. This
study used simple BERT without fine-tuning in the specificdomain of movie reviews.
There are also examples of BERT applied to information
retrieval. Yang et al. [7] proposed a method for adapting
BERT to the ad hoc retrieval of documents. Experiments withTREC Microblog Tracks showed that BERT-based methods
are effective for retrieving short documents, such as thosefound in microblogs. Yunqiu et al. [8] proposed a BERT
model for legal case retrieval, BERT-PLI, that can retrieve
from much longer queries than general queries. A related
precedent retrieval task using the COLIEE 2019 dataset revealed that this method can more accurately understand themeaning of longer documents. Zhuolin et al. [9] proposed a

Keyword
Vectorization

Input
Tearjerker

Vectorizing
with BERT

Movie Vectorization
by Review
Titanic“Terminator 2”

Terminator 2

BERT

Keyword QueryI needed a handkerchief.

“Titanic”

BERT

The last scene made me crying“Tearjerker”

Dancer in the Dark

Vector of
KeywordQuery“Dancer in the Dark”

Vector ofMovies(Created from Reviews)

Vectorizing
with BERT“Hachi”

Trained
Nural
Network

0.8“Tearjerker”

0.7

Nural
Network

Used for
Ranking

Probability

0.9

Is "Titanic" likely to appear in a
User Generated Ranking withthe title containing “Tearjerker"?

saddest Hachi

Training

funnyToy Story

This combination
is often seen
within User
Generated Ranking.

"My top 10 saddest movies"

Vectorizing
with BERT

saddest Titanic

1. Titanic
2. Hachi"Best Funny Movies Ever"

1. Home Alone
2. Toy Story･･･

Ranking of All Movies“Titanic”

Vectorizing
with BERT

ConvertTrainedModel

funny Home Alone･･･“Tearjerker”“Home Alone”

0.9

Inference･･･“Tearjerker” “Titanic”

Concatenate

Review Site

Training of Relevance Model

Relevance Calculation

Output“Tearjerker”

Toooo sad !!!

User Generated Rankings

Fig. 1. Overview of our proposed method. Estimating the relevance between the query and movies using the user-generated ranking as the correct answer.
When the model inputs a vector of queries and an arbitrary movie, it outputs the probability that the combination exists in the correct answer dataset.

method for retrieving multilingual documents from English
queries by using BERT to learn the relationships between
English queries and multilingual documents. The task ofretrieving Lithuanian text and audio documents using the queryand search corpus provided by IARPA’s MATERIAL program
revealed that this method enables more accurate retrieval thanother methods.
Since this research was conducted on movies, which are
visual images, it is difficult to deal with the contents of movies
in text; therefore, we used user-posted reviews. In addition, it
is difficult to compare the similarity of review sentences andshort queries because of the different natures of the sentences.
Therefore, we used Learning to Rank to match queries andreview sentences.
C. Learning to Rank
Learning to Rank is a method for solving ranking problems
using machine learning. It is also called ranking learning.
Three main approaches to Learning to Rank [10] are pointwise, pairwise, and listwise methods. In this study, we used
the pointwise method. The Learning to Rank method aims torank highly relevant documents in response to a query, and it
is mainly used for information retrieval.
As an example of retrieving documents belonging to a
specific topic, Amir et al. [11] proposed a method that used
BERT and Learning to Rank to retrieve evidence to support a
claim. The task of retrieving sentences that provided evidence
using the FEVER dataset showed that this method performed
best compared to other methods.
Yu et al. [12] proposed a method using Learning to Rank
to find documents containing answers to a given question.

Experiments on a standard TREC benchmark dataset showedthat the method’s performance was comparable to state-of-theart methods used to retrieve answer sentences. Fabio et al. [13]proposed a method using Learning to Rank for content-based
image retrieval. Evaluation experiments on two datasets, CorelGALLERY Magic Stock Photo Library 2, and Caltech, showedthat the method outperformed traditional ranking methods.
Since the current research is concerned with the retrieval ofan item (i.e., a movie), we also discuss some examples ofapplying Learning to Rank to item retrieval.
Shubhra et al. [14] proposed a method that applied
Learning to Rank for product retrieval on an e-commercesite. They found that the method, which used LambdaMART,a ranking learning method, provided the highest score fora search task using randomly selected queries and productinformation from the web. Blake et al. [15] proposed a method
using Learning to Rank to improve the accuracy of location
retrieval using inaccurate GPS data. Experiments using actual
check-in information about facilities in Manhattan showed that
this proposed method had the highest accuracy for locating
locations.
Prior to this study, Kurihara et al. [16] proposed a method
using Learning to Rank for movie retrieval based on review
information. They proposed a model that estimated the ranking
order of movies using a decision tree-based method called
LambdaMART. However, the current research aimed to makeit possible to rank movies in a new way by learning therelevance of queries and items using a neural network.

III. M ETHOD
In this study, we propose a search algorithm that ranks
movies in response to an arbitrary keyword query. An overview
of the method is shown in Figure 1. The method first extracts
the titles and movies included in the rankings from usergenerated movie rankings. Next, both the words and movies in
the ranking titles are vectorized using BERT. The two vectors
are then concatenated and input into neural networks, whichlearn them as a binary classification problem. That is, does amovie appear in the rankings with this keyword in its title?
By inputting an arbitrary query and movie into the learned
model, it is possible to estimate how likely the movie willappear in the rankings that include the query as a title. By
calculating the probability of appearance for all movies, amovie ranking can be generated from an arbitrary query.
A. Vectorizing Movies Using Reviews
Movie websites contain movie reviews posted by various
users. In this study, it was assumed that user-posted reviews contain information that represents the movie’s features.
Therefore, when representing a movie as a vector, a review
sentence for that movie was used instead of a movie’s metadataor visual information.
First, the text was preprocessed to vectorize the review
sentences. Since there was a limit to the number of tokens
that BERT could vectorize, it was impossible to vectorize long
review sentences. Therefore, the review sentences were split
by punctuation marks, question marks, exclamation marks, andother symbols. Unnecessary characters and symbols were also
removed.
Next, each preprocessed reviewed sentence was input into
BERT to calculate a 768-dimensional vector in a distributed
representation format. However, BERT could not convert longsentences into distributed representations due to the upper
limit of computational complexity. Therefore, each sentencer ∈ R(m) in the movie review m was vectorized and treatedas a feature vector of the movie by pooling it. When thevector representing any review sentence r was denoted by
BERTr (r), the vector v(m) of movies m was defined asfollows:Pr∈R(m) BERTr (r).
(1)v(m) =|R(m)|
This is the average of the vectors of all the review sentences
for that movie.
B. Formatting User-Generated Rankings as Training Data
To enable movie retrieval for a specific query, correct
training data linking the query to the movies was required.
Therefore, this study used data from websites where users
can submit their own movie rankings as the correct data. We
collected user-generated rankings registered in Yahoo! Movies
Japan for our experiment.
In recent years, sharing arbitrary movie rankings createdby individual users on the Internet has become common. For
example, it is easy to find rankings such as “My personal top
10 sad movies” on personal blogs. There are also many webservices that enable users to post such rankings. For example,
IMDB, the world’s largest movie review site, provides afunction called Watchlist1 .” This feature enables users to createa list of arbitrary titles and register their favorite movies on
the list. In Japan, Yahoo! Movies’ Round-Up enables users tocreate a list with an arbitrary title and register up to 10 movies.
Many of these individually created rankings include titles suchas “The 10 best movies that make me cry” or “The 10 best
comedy movies I like.” In this study, the model learned therelationship between queries and movies by using these kindsof user-generated rankings as correct answer data.
The data were not clean because the general public creates
these user-generated rankings. They contain many irrelevant
descriptions and miscellaneous rankings and therefore require
preprocessing. First, we excluded rankings that were not basedon a specific point of view. These rankings vary widely; someusers simply rank the movies they like. For example, rankings
such as “10 Favorite Movies” or “My Top 10 Movies of2002” do not contain a specific point of view. We manuallycreated dictionaries to eliminate such rankings. We eliminatedrankings that included years or specific words, such as “alltime.” Words that were unlikely to be connected to the query
were then removed from the ranking titles. Specifically, thesewere words such as “my,” “top,” and “movie,” and adverbs.
Next, the training data were formatted to train the model.
The training data consisted of input data and a correct answer
label. The title of the ranking and the movies in it were
vectorized using BERT. Each pair of movie and ranking title
vectors was combined and used as input to the neural network.
The correct answer label was a binary value of 0 or 1,
indicating whether the movie was included in the ranking.
Positive examples were generated using the movies includedin the ranking. Negative examples were randomly generated
from movies that were not included in the ranking.
C. Learning the Relationship Between Movies and Terms in
the Ranking Title
Relevance between a query and a movie is computableby considering the ranking title as a keyword query. Here,a simple neural network was used to calculate relevance.
When each movie and query has been represented as a vector
of distributed representations, the proposed method trains aneural network as a binary classification task using these
vectors, as shown in Figure2. The task is to combine vectorsof movies and queries and then estimate whether the movie
will likely appear in a ranking that includes the query in its
title.
The input was a 1,536-dimensional vector: a combined pairof a 768-dimensional vector representing a movie and a 768dimensional vector representing a ranking title. This neuralnetwork consisted of four full-combination layers, as shownin Table I. The output layer was binary since it performs abinary classification of whether a movie appears in the ranking
whose title contains a given query.
1 IMDB Watchlist: https://imdb.com/list/watchlist

we listed the movies that appeared in the rankings generated
by each method and asked the participants to evaluate how
well they matched the query.

Estimation

Is the movie “Titanic”
included in theuser-generated ranking
named “tearjerker”?

A. Compared Methods“Titanic”“My top 10 tearjerker movies”

Fig. 2. Procedure for training the model to predict whether or not amovie will appear in user-generated rankings that include the queryin the title.
TABLE ID ETAILS OF THE NETWORK LAYER USED FOR THE TRAINING .
Layer Type# Nodes

Activation
Functionfully connected
fully connected
fully connected
fully connected

1,536
64
64
2

Relu
Relu
Relu
Sigmoid

The loss function loss(p, q) is expressed as
Xloss(p, q) = −p(x)log(q(x)),(2)

xwhere p is the value of the correct answer, and q is thepredicted value. The values obtained by this loss function wereused to optimize the network model.
D. Generate Ranking for a Given QueryUsing the language model learned this way, movies wereranked based on arbitrary queries. The current method calculated the relevance of a query for all movies in a round-robinfashion, and it ranked movies in order of relevance.
The incoming query was vectorized using BERT. The vector
was then combined with the vector of a certain movie. These
combined vectors were input into the trained model createdin section III-C. This calculated the relevance of the query to
the movie. This process was then repeated for all movies ina brute-force manner. Since the relevance to all movies wascalculated, it could rank the movies for the query.
IV. E VALUATION
A participant experiment was conducted to verify whether
the movie ranking generated by the proposed method wasconsistent with the user’s perception. For a particular query,

Two hypotheses that can be linked to the technical contribution of this study are as follows:
H1 : Review information can be used to search for movies
by keywords that do not appear in the movie synopsis or
metadata.
H2 : If each of the documents and queries about a movie isrepresented as a vector, a simple similarity comparison,such as cosine similarity, is not sufficient.
To test these hypotheses, we compared three variant methods and the proposed methods, as follows:• Proposed Method: A method for matching movie vectors
generated from reviews with queries using deep learning.
• Movie Similarity: A method for comparing the similarity
between the movie vector generated from the reviews andthe query vector.
• Review Sentence Similarity: A method for comparing
the similarity between the review sentence vector and thequery vector.
• Metadata Only: A method using vectors of the movie’s
metadata, such as description text on movie sites.
Proposed Method uses deep learning to estimate the relevance of the query vector to the vectors generated from themovie reviews described in Section III. The method reflects
both hypotheses H1 and H2.
Movie Similarity compares the vectors generated from themovie reviews and the query vectors using cosine similarity.
It takes a query as input and vectorizes the query using BERT.
Then, the cosine similarity between the query vector and themovie vector is calculated. Based on the results, movies withthe highest similarity were output as a ranking. This method
reflects only hypothesis H1.
Review Sentence Similarity compares the similarity between the review vectors used to generate the movie and query
vectors. This calculates the cosine similarity between the vector of input queries and the vector of review sentences posted
on Yahoo! Movies with highly similar reviews are ranked
based on the similarity between the review text and the query.
When multiple reviews posted for the same movie appear in
the ranking, the similarity of the most similar review is treatedas the movie’s similarity. This method reflects hypothesis
H1. However, hypothesis H2 is based on the assumption thatsuperficial similarity is insufficient for comparing a query toa movie. Thus, it corresponds to a review sentence to a query
instead of reaching a movie to a query.
Metadata Only uses only metadata and no reviews. The
information about a movie includes metadata, such as theoriginal title, production year, running time, production country, genre, director, executive producer, script, music, etc., andtext such as commentary, synopsis, etc. First, text matching is
performed between the input query and the above information.

TABLE II
Q UERIES USED IN THE EXPERIMENT AND THEIR FEATURES
Query
Tearjerker
Laughable
Shocking
Suitable for dating
Suitable for children
Suspense
Animation
Ghibli
Surprise ending
Takeshi Kitano

TABLE IIIP RECISION AT k AND N DCG FOR EACH METHOD

Features
Emotion after watching the movie
Situation or scene when watching the movie
Category of the movie

Proposed Method
Metadata Only
Movie Similarity
Review Sentence Similarity

p@1
0.50
0.50
0.20
0.80

p@5
0.56
0.46
0.24
0.74

p@10
0.58
0.40
0.27
0.72

nDCG
0.64
0.60
0.47
0.75

Content of the movie

library for neural networks, was used for implementing the
ranking models.
If there are any matched movies, the cosine similarity between
the query vector and the vectors of the commentary andsynopsis sentences of those movies is calculated. Movies withhigh similarity are ranked in the output. This method does notsupport either hypothesis H1 or H2.
B. Dataset
We collected about 15,000 movies with 10 or more reviews
from a certain real movie review site (anonimized). We collected about 40,000 movie reviews and about 10,000 usergenerated rankings from Round-Up, Yahoo! Movies as correct
answers for training. We used about 7,000 of these data forour research, excluding rankings that did not include specificviewpoints.
C. Experimental Tasks
A subject experiment was conducted online in which participants were asked to label the relevance between the query andthe movie. Ten participants performed the labeling. First, participants were given a link to a premade Google spreadsheet.
Each sheet contained a query with up to 40 movie titles, one
for each row. This is a mixture of the top 10 movie rankings
output by each of the four methods for a single query. The
participants rated how closely each movie related to the queryon a 5-point scale from one to five. Participants were allowed
to search for movie information on the Internet if they were
not familiar with the movie.
We prepared 10 queries in advance. The actual queries areshown in Table II. Two participants labeled 80 movies for
two queries.
D. Implementation
We implemented a system that could calculate the relevance
between a given query and movies for each of the comparison
methods. The system inputs a query and output the top 10
rankings of each method. The BERT Japanese pretrainedmodel2 was used as the BERT model to vectorize the query.
MeCab3 , a famous Japanese morphological analyzer was usedas the tokenizer for the BERT vectorization. As a vocabulary
dictionary for MeCab, mecab-ipadic-neologd4 , which supports
new words and proper expressions, was used. Keras, a Python
2 Kurohashi Lab. Kyoto University: https://nlp.ist.i.kyoto-u.ac.jp/EN/
3 MeCab: Yet Another Part-of-Speech and Morphological Analyzer:
https://taku910.github.io/mecab/
4 mecab-ipadic-NEologd: Neologism dictionary for MeCab
https://github.com/neologd/mecab-ipadic-neologd

E. Experimental Results
This section describes the results of the subject experiment.
We compared the methods by using three metrics: P@k for the
precision of each method, nDCG (normalized Discounted Cumulative Gain) for the ranking accuracy, and the participants’
average rating.
First, the top part of the ranking of each method wasevaluated according to precision. Precision is a measure of how
correct the answers were included at the top of the ranking. In
this experiment, a rating of three or higher on the participant’s
5-point scale was considered correct, and the accuracy wascalculated based on this number of correct answers. p@k(precision at k) represents the precision of the movies up to
the k position in the ranking.
The precision for each method is shown in Table III. Review
Sentence Similarity was the most accurate p@1, p@5, andp@10. The proposed method was ranked second for all p@1,
p@5, and p@10. Movie Similarity had the lowest accuracy
for all p@1, p@5, and p@10.
Next, nDCG was used to check the accuracy of the ranking.
The nDCG for each method is shown in Table III. The mostaccurate method was the Review Sentence Similarity, with ascore of 0.75. In contrast, the Movie Similarity method had
the lowest nDCG score of 0.47.
F. Ratings by Participants for Each QueryIn this experiment, participants were asked to rate on a 5point scale how closely a movie related to a given query. The
average of the ratings by the participants for each method
and each query is shown in Table IV. For the queries “Tearjerker,” “Suspense,” “Suitable for dating,” and “Laughable,”the proposed method was the most accurate in finding thehighest-rated movies. For the queries “Anime,” “Shocking,”and “Takeshi Kitano,” the method using Metadata Only was
the most accurate. For all other queries, the Review Sentence
Similarity method was the most accurate.
As an example of the output rankings, Table V shows the top
10 movies for the query “tearjerker” by the proposed and thebaseline methods. In this result, participants labeled all movies
found by the proposed method as “tearjerker” movies (i.e.,
all movies were rated at least three). The proposed method
seemed to be well-suited for searches based on the viewer’s
impression of the movie as a whole.
In contrast, as an example of the case of the proposed
method not working well, Table VI shows the ranking for theTABLE IV
AVERAGE PARTICIPANTS ’ RATING OF THE MOVIES IN THE TOP 10 RESULTS FOR QUERIES (5- POINT SCALE , 1 TO 5)

Proposed Method
Metadata Only
Movie Similarity
Review Sentence Similarity

Tearjerker

Animation

Suspense

Ghibli

3.7
2.1
2.1
2.9

2.2
3.8
2.2
2.6

3.6
3.1
1.7
3.6

1.6
1.9
1.7
4.3

Suitablefor dating
3.5
1.9
2.9
3.5

Suitable forchildren
3.8
1.7
2.7
4.2

Shocking
2.4
3.7
2.4
3.4

Surpriseending
3.9
2.9
2.9
4.5

Takeshi
Kitano
1.5
4.0
1.8
3.6

Laughable
3.0
1.7
1.7
1.7

TABLE VE XAMPLE OF RESULTS THAT THE PROPOSED METHOD WORKED WELL ( FOR THE QUERY “T EARJERKER ,” RATING BY PARTICIPANTS WAS ON A 5- POINT
SCALE , 1 TO 5.)
Proposed Method
Movie title
What Dreams May Come
Gray Sunset
Glory Daze
The Boy Who Could Fly
Crayon Shin-chan: The Adult Empire Strikes Back
Pay It Forward
It’s a Wonderful Life
Jack
Life is Beautiful
The Notebook

query “Takeshi Kitano” (note that because both the query anddataset were in Japanese, many Japanese movies were includedin the search results). In this case, the Metadata Only method
was highly accurate, but methods using review did not work
well. The proposed method found many animation movies not
related to the query.
V. D ISCUSSION
This section discusses the experimental results, the effectiveness of the methods, and which methods are suitable foreach query and task. Overall, the proposed method was moreaccurate for retrieval than the cosine similarity or metadata
methods. However, a simple similarity calculation between asingle sentence in a movie review and a keyword query waseven more accurate.
One possible cause of this was the limitation of representing
the entire movie by a single vector. If all reviews for a movie
were vectorized and summarized by pooling, the overall trendof the movie (i.e., sad, funny, and warm) might remain as a feature. However, individual scenes and minor impressions would
be diluted when summarizing many opinions. Even underthese conditions, the proposed method was more accurate thansimple cosine similarity. The vector properties were different
between a vector of short keyword queries and a vector of
longer reviews from multiple people. Therefore, the methodof calculating the validity of the combination using a neuralnetwork would have been effective, since simply taking thecosine similarity would not correctly calculate the similarity.
The method of Review Sentence Similarity is considered tobe more accurate because a highly granular variance representation can be obtained from the review sentences. The nDCG
had the highest Review Sentence Similarity, confirming theeffectiveness of using reviews. This was probably because, in
our experimental search task, it was more important whether

Rating
3.0
3.5
3.0
4.0
4.5
3.5
4.0
3.5
4.0
4.0

Movie Similarity
Movie title
PPiL GU
Yellow Hair
Columbo Goes to the Guillotine
Lightereul kyeora
Nan va Koutcheh
New Deka Matsuri
Human Traffic
JUNGLE JUICE
Rainy Dog
Bad Reputation in the Marketplace

Rating
2.0
1.5
2.0
1.5
2.0
1.5
3.0
2.0
3.0
2.0

there was an element applicable to the query than the overall
trend of the movie. For example, whether the movie has asurprise ending depends not on the movie as a whole, but on asingle scene. In this case, it was possible to determine whether
the review included a description of the turnover scene without
reading reviews for the entire movie. In such a case, it would
simply be more accurate not to characterize the entire movie.
The scores for each query indicated that the effective
methods varied depending on the query type. The metadatabased methods scored higher for queries for which there was
likely to be information in the metadata, such as “Animation,”“Shocking,” and “Takeshi Kitano.” However, the review-based
method scored higher for queries that describe the nature of
the movie, such as “Tearjerker,” “Laughable,” “Suitable fordating,” and “Surprise ending.” This shows that using reviews
makes it possible to search for movies based on informationthat does not exist in the metadata.
VI. C ONCLUSION
This paper proposes a method for calculating the relevance
between a certain keyword query and movies based on usergenerated rankings and movie reviews. The model learned therelationship between words in the ranking title and movie
reviews using neural networks that used data taken from auser-generated movie ranking forum. Learning through thetask of estimating whether a movie is included in the ranking
with a given title enabled the model to rank movies.
To verify the usefulness of the model, we conducted subject
experiments. Participants were asked to score the relevance
between the given queries and movies ranked high by eachcomparison method. Experimental results showed that the proposed method, which uses user-generated reviews, was moreaccurate than traditional simple cosine similarity; however, it
was even more accurate to split reviews into sentences andTABLE VI
E XAMPLE OF A CASE WHERE THE M ETADATA O NLY METHOD WORKED WELL ( FOR THE QUERY “TAKESHI K ITANO ,” RATING BY PARTICIPANTS WAS ON
A 5- POINT SCALE , 1 TO 5.)
Proposed Method
Movie title
Crayon Shin-chan: Unkokusai’s Ambition
Case Closed: Captured in Her Eyes
Godzilla, Mothra and King Ghidorah
Air
Lupin III: Hemingway Papers
Iki-jigoku
MosquitoStar Wars: Episode V
RahXephon
Lupin III: The Castle of Cagliostro

use the maximum similarity method for each of them. We also
found that for some tasks, using metadata was more accurate.
We plan to use the proposed model to determine whether
a given movie appears in a given titled ranking or not.
Specifically, we are considering using this as a fine-tuningtask for the language model.
ACKNOWLEDGMENTS
This work was supported by JSPS KAKENHI Grants Numbers 21H03775, 21H03774, and 22H03905.
R EFERENCES[1] T. Miyashita, Y. Shoji, S. Fujita, and M. J. Dürst, “Movie keyword
search using large-scale language model with user-generated rankingsand reviews,” in The 25th International Conference on Information
Integration and Web Intelligence, 2023, pp. 249–255.
[2] J. Ramanand, K. Bhavsar, and N. Pedanekar, “Wishful thinking - finding
suggestions and ’buy’ wishes from product reviews,” in Proc. of the
NAACL HLT 2010 Workshop on Computational Approaches to Analysis
and Generation of Emotion in Text, 2010, pp. 54–61.
[3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pretraining of deep bidirectional transformers for language understanding,”
in Proc. of the 2019 Conference of the North American Chapterof the Association for Computational Linguistics: Human Language
Technologies, 2019, pp. 4171–4186.
[4] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings
using Siamese BERT-networks,” in Proc. of the 2019 Conference on
Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP), 2019, pp. 3982–3992.
[5] Z. Liu, D. Huang, K. Huang, Z. Li, and J. Zhao, “Finbert: A pre-trained
financial language representation model for financial text mining,” inProc. of the Twenty-Ninth International Joint Conference on Artificial
Intelligence, IJCAI-20, 2020, pp. 4513–4519.
[6] W. Sakata, T. Shibata, R. Tanaka, and S. Kurohashi, “Faq retrieval using
query-question similarity and bert-based query-answer relevance,” inProc. of the 42nd International ACM SIGIR Conference on Researchand Development in Information Retrieval, 2019, pp. 1113–1116.
[7] W. Yang, H. Zhang, and J. Lin, “Simple applications of bert for ad hoc
document retrieval,” arXiv preprint arXiv:1903.10972, 2019.
[8] Y. Shao, J. Mao, Y. Liu, W. Ma, K. Satoh, M. Zhang, and S. Ma, “Bertpli: Modeling paragraph-level interactions for legal case retrieval.” inIJCAI, 2020, pp. 3501–3507.
[9] Z. Jiang, A. El-Jaroudi, W. Hartmann, D. Karakos, and L. Zhao, “Crosslingual information retrieval with BERT,” in Proc. of the workshop
on Cross-Language Search and Summarization of Text and Speech(CLSSTS2020), 2020, pp. 26–31.
[10] T.-Y. Liu, “Learning to rank for information retrieval,” Found. Trends
Inf. Retr., vol. 3, no. 3, pp. 225–331, 2009.
[11] A. Soleimani, C. Monz, and M. Worring, “Bert for evidence retrieval
and claim verification,” in Advances in Information Retrieval, J. M.
Jose, E. Yilmaz, J. Magalhães, P. Castells, N. Ferro, M. J. Silva, andF. Martins, Eds., 2020, pp. 359–366.

Rating
1.0
1.0
1.5
2.0
1.5
2.0
1.0
1.0
2.0
1.5

Metadata Only
Movie title
Rating
Kikujiro
4.5
Sonatine
5.0
Boiling Point
3.5
Zatoichi
4.0
Beyond Outrage
4.0
Outrage Coda
4.0
BROTHER
3.5
TAKESHIS’
4.5
HANA-BI
3.0
OUTRAGE
4.0[12] L. Yu, K. Hermann, P. Blunsom, and S. Pulman, “Deep learning for answer sentence selection,” Proc. of the Deep Learning and Representation
Learning Workshop: NIPS-2014, 2014.
[13] F. F. Faria, A. Veloso, H. M. Almeida, E. Valle, R. d. S. Torres,
M. A. Gonçalves, and W. Meira, “Learning to rank for content-based
image retrieval,” in Proc. of the International Conference on Multimedia
Information Retrieval, 2010, pp. 285–294.
[14] S. K. Karmaker Santu, P. Sondhi, and C. Zhai, “On application of learning to rank for e-commerce search,” in Proc. of the 40th International
ACM SIGIR Conference on Research and Development in Information
Retrieval, 2017, pp. 475–484.
[15] B. Shaw, J. Shea, S. Sinha, and A. Hogue, “Learning to rank for spatiotemporal search,” in Proc. of the Sixth ACM International Conferenceon Web Search and Data Mining, 2013, pp. 717–726.
[16] K. Kurihara, Y. Shoji, S. Fujita, and M. J. Dürst, “Learning to rankbased approach for movie search by keyword query and example query,”in The 23rd International Conference on Information Integration and
Web Intelligence. New York, NY, USA: Association for Computing
Machinery, 2021, pp. 137–145.