DEIM Forum 2019 G6-2

Query Keyword Extraction from Complaint Data for Collecting Advice
Liang YANG† , Daisuke KITAYAMA†† , and Kazutoshi SUMIYA†† Kwansei Gakuin University 2-1 Gakuen, Sanda-shi, Hyogo, 669-1337 Japan†† Kogakuin University 1-24-2 Nishi-Shinjuku, Shinjuku-ku, Tokyo, 163-8677 Japan
E-mail: †{dui93794,sumiya}@kwansei.ac.jp, ††kitayama@cc.kogakuin.ac.jp
Abstract Nowadays, there are a large number of users who post complaints about a certain service on the Internet. Because users have various values and views, even if they receive the same service, they may complain in
diﬀerent ways. However, it is quite diﬃcult to respond to various user demands for service in real time and there arealmost no direct solutions when users feel dissatisfied with a certain service. Therefore, in this paper, we propose
a method that generates query keywords according to various user complaints about a certain service for collecting
advice. To generate query keywords, we extract the company names, complaint topic words and advice topic words
from complaint data. Then suitable web pages containing advice are recommended from the results of the query.
This advice could address users’ dissatisfaction and respond to their diﬀerent demands in a comprehensive way.
Key words query extraction, complaints, advice.

1. Introduction

company, we extract complaint topic words that represent
users’ dissatisfactions. Third, we extract advice topic words

Nowadays, there are a large number of users who post com-that can help users to collect advice according to their com-plaints about a certain service on the Internet. Because users

plaint. Finally, we generate the query based on the com-have various values and views, even if they receive the samepany name, complaint topic word, and advice topic word.

service, they may complain in diﬀerent ways. However, itMoreover, we use an OR-based search to acquire suitable

is quite diﬃcult to respond to various user demands for ser-web pages containing advice by using the result of the query.

vice in real time and there are almost no direct solutions

This advice could address users’ dissatisfaction and respondwhen users feel dissatisfied with a certain service. What’s

to their diﬀerent demands in a comprehensive way.

more, it is diﬃcult to determine the expected solutions if

The remainder of this paper is structured as follows. Sec-the query keywords are not appropriate, especially for cer-tion II presents a brief summary of related work. Section

tain users who are not used to online search, like the elderly

III introduces the dataset we use for research and explains

and foreigners. For instance, users might complain about an

the proposed method. Section IV discusses the experimental

online retailer’s cardboard box using diﬀerent phrases such

results and the evaluation of the proposed method. Finally,

as “the size is too big,” “low quality,” or “over-wrapped.”

Section VI concludes this paper and discusses future work.

It is diﬃcult to resolve users’ complaints in a direct way because items such as a cardboard box are chosen by retailers

2. Related Workand users cannot determine their size or quality by them-

2. 1 FKC dataset

selves. Moreover, the expected solutions would not be found

The FKC dataset has been used for several studies in re-

if users search with keywords such as “cardboard box, toocent years. Mitsuzawa et al. [1] presented the FKC dataset

big.” Hence, we suggest resolving users’ complaints in an

which is from Fuman Kaitori Center (FKC). Fuman means

indirect way by oﬀering them advice like “how to modify adissatisfaction in Japanese. The FKC is a Japanese con-cardboard box” or “how to utilize a cardboard box for stor-

sumers’ negative opinion data collection and analysis service.

age.”

In our work, we used and analyzed the FKC dataset.

To oﬀer these kind of advice, in this paper, we propose

Hasegawa et al. [2] analyzed and visualized the contents

a method that generates query keywords according to var-of the FKC dataset such as the distribution of users’ ages,

ious user complaints about a certain service. To generate

jobs, and gender. In our work, we determine the aims of thequery keywords, we first analyze complaint data from theexperiment based on their results.

Fuman Kaitori Center website and extract company names

Hayashi et al. [3] proposed a system to recommend appro-

from the data. Second, from the negative reviews for each

priate products for users according to their complaints. This

Figure 1 System Flow

recommendation could directly resolve users’ dissatisfaction

query keywords from negative reviews to recommend web

with certain products. In our work, we propose a method to

pages of advice with the aim of addressing a user’s dissatis-

resolve user complaints about services instead of products infaction with a certain service.

an indirect way.
2. 2 Topic Word Extraction
Sakai et al. [5] proposed a method to extract negative

3. Proposed Method
3. 1 Overview

words as the expressions of dissatisfaction from blogs. TheyIn this paper, we propose a method to generate query

extracted nouns, adjectives to make a dissatisfaction expres-keywords for users to search for advice according to theirsion dictionary. In our work, we only extract nouns because

various complaints about a certain service. Figure 1 shows

nouns can explain and represent the content of users’ com-the system flow of our proposed method. First, we extract

plaints.

the company name and complaint topic words by calculat-

Hashimoto et al. [6] proposed a method to extract impor-

ing the importance of the nouns in the negative reviews.

tant topics from newspaper and detect social problems based

Second, we obtain candidate search keywords of these ex-on document clustering. Ustumi et al. [4] proposed a method

tracted words. Then, the system extracts the advice topic

to extract technological solutions to social problems such as

word by calculating the relevancy of the candidate keywords

medical issues from the news. They extracted technologi-

to the FKC dataset and score them using morphological andcal solution words by calculating the relevance of problems

polarity analyses. Third, we create the query by combining

and technologies. They defined the relevance calculation as

the company name, complaint topic word, and advice topic

problem relevancy and technical relevancy. A higher value ofword. These query keywords could respond to the needs andrelevancy indicated a higher possibility of being able to ex-

search intentions of diﬀerent users in a comprehensive way.

tract a technological solution word. In our work, we use thisFinally, suitable web pages containing advice that could ad-concept and extract the advice topic word by calculating the

dress a user’s complaint are recommended from the results

relevance of the company and complaint topic. However, weof the query.

hypothesize that a lower relevancy indicates a higher proba-

3. 2 Dataset

bility that a word is an advice topic word.

In this study, we analyze a dataset of complaints from the

Yoshida et al. [7] proposed a method to extract features

Fuman Kaitori Center, which is provided by Insight Techterms from the customer reviews of e-commerce sites in or-

Inc. from the National Institute of Informatics. In this pa-

der to recommend similar items to users. They used po-

per, we refer to the Fuman Kaitori Center’s dataset as thelarity analysis to calculate the degree of importance of fea-

FKC dataset. The Fuman Kaitori Center is a website on

ture words by counting the number of positive reviews, neg-

which users can post their complaints about topics such asative reviews, and positive ratings. In our work, we also use

products, services, education, work, and relationships. More-polarity analysis to evaluate advice topic words. However,

over, users get points when they post complaints that they

we weight words according to the result of polarity analysis.
2. 3 Query Generation

can exchange for coupons for online shopping websites. ThisSong et al. [9] and Kajinami et al. [10] proposed a sys-

posted from 18 March 2015 to 12 March 2017 by aroundtem to generate query keywords that can support a user’s

100,000 users. Each negative review contains the informa-search intention. Kakimoto et al. [8] proposed a system totion shown in TABLE I. For example, the FKC dataset con-extract query keywords from the closed caption data of TV

tains categories like “public transportation,” “restaurants,”

programs to recommend web pages related to tourism andand “industry.” Each category contains several subcategories

events based on users’ preferences. In our work, we extractthat describe the detailed information of the category. For in-dataset contains about 5 million negative reviews that werestance, the category “industry” contains subcategories such“company relevancy” as the relevance between a candidate

as “agriculture,” “architecture,” and “IT web services.” In

keyword and a particular company. It is calculated using theaddition, each subcategory contains several companies that

following equation.

are recorded in the dataset. Moreover, because we focus on
user service complaints, we only analyze the data fields for“company” and “text” in the method proposed in this paper.

company relevancy =

Rcd
Rc

(2)

Here, Rcd is defined as the number of occurrences of cer-

Table 1 Details of the FKC dataset

tain candidate keyword in complaints for the company in the

Data item

Content

FKC dataset and Rc is defined as the number of negative re-post idcomplaint ID

user id

Fuman Kaitori Center ID

category

complaint category

subcategory

detailed complaint categorycompany

company nameproduct

product nametext

negative review

views of that company.
Next, we define the relevance between a candidate keyword and a complaint topic word as the “complaint topic
relevancy” using the following equation.
complaint topic relevancy =

Rtd
Rt

(3)

Here, Rtd is defined as the number of occurrences of thecandidate keyword with the complaint topic word in the neg-

3. 3 Extraction of Company Names and Complaint Topic Words

ative reviews of the FKC dataset and Rt is defined as thenumber of negative reviews with that complaint topic word.

Our proposed method extracts company names from FKC

However, not all candidate keywords with low relevance indataset directly from the company field of each record. Next,

the FKC dataset can be used as advice topic words. For ex-

we extract the complaint topic word by analyzing negative

ample, some negative words as well as verbs and adjectives

reviews. In this paper, we only use the negative reviews that

do not help users acquire advice. Hence, it is necessary to

are labeled with the company name. To extract complaint

exclude these kinds of candidate keywords. To do this, we

topic words, we first extract all companies’ negative reviews

weight candidate keywords using morphological and polarity

for one subcategory. Then, we use morphological analysis

analyses, as shown in TABLE II.

to extract all nouns from the negative reviews. Next, weTable 2 Weights for Candidate Keywords

calculate the importance of each noun using the following
equation.
tf
tf
×∑
|A|
tfd
d∈D

(1)

Here, tf is defined as the number of occurrences of a par-Result of Analysis

Weightnegative

0.8

verb

0.7

adjective

0.7

proper noun (place name)0.7

ticular noun in the complaints for a certain company, |A| isproper noun (organization name)0.3

defined as the number of all nouns in the complaints for acommon noun

0.3

certain company, andverbal noun

0.1∑

tfd is defined as the number ofd∈Doccurrences of certain noun for the complaints for all companies. After calculating the importance of each noun, we
determine the threshold value for that company. Finally,
we extract all nouns whose importance values are above thethreshold and define them as that company’s complaint topic
words.

Finally, we calculate the final score of the candidate keywords by combining the arithmetic mean of the company
and complaint topic relevancies with the weight as the following equation. Here, relevancies are calculated by adding
the results of equation 2 and 3.

3. 4 Extraction of Advice Topic Words
To extract the advice topic word, we first obtain candidate
search keywords from a Google search of the company nameand complaint topic word. Because the FKC dataset is fullof negative reviews, we hypothesize that candidate keyword
that are less relevant to the complaint data will make betteradvice topic words. To verify this hypothesis, we calculated
the relevance of these candidate keywords for each company
and each complaint topic word. In this study, we define theScore =

relevancies
× W eight
2(4)

After calculating the final score of each candidate keyword,
we determine the threshold value for each company. Candidate keywords those scores are under the threshold value
become the advice topic words.

Table 3 Example Complaint Topic Words

3. 5 Generation of web search queries
To respond to the diﬀerent demands of users, each com-

Complaint topic word

Companypany name and complaint topic word are matched with sev-

prime, delivery, movie,Aeral advice topic words. To search for suitable websites, We

cardboard box, delivery fee, order,

use an OR-based search method to acquire advice websites.

reservation, book, print, return

Our proposed system generates the query based on one com-stamp, code, block, coinBpany name, one complaint topic word, and one advice topic

backup, telephone call, camera, setting,word.

post, commercial

3. 6 Recommendation of Advice

news, question, premium,C

Our proposed system recommends suitable web pages con-answer, auction, article, title,

taining advice from the results of the query which is based

ID, weather forecast, transaction

on users’ complaints. Figure 2 shows the user interface of our
proposed system. First, the system generates several queries

Next,we extracted advice topic words from the candidateby analyzing user’s negative review. Next, When user clicks

keywords that had a score less than 0.0043, 0.0020, andon one of the oﬀered query, the query will turn into the web

0.0033 for companies A, B, and C, respectively. We extracted

page of the results of search. Finally, user can choose andadvice topic words that match with 20, 10, and 10 complaintbrowse the web page based on their needs. The system shows

topic words for companies A, B, and C, respectively. Somethe advice that could address user’s dissatisfaction expressed

examples of these words are shown in TABLE IV. As TA-

in the negative review.

BLE IV shows, even if the candidate keywords are under thethreshold value, not all of them can be used as advice topic
words. However, the proposed method is suﬃcient for ranking candidate keywords. Most of the extracted results can beused as advice topic words. In the future, we plan to develop
a method to determine a more accurate threshold value tobetter exclude noise in the results.
Table 4 Example Advice Topic Words

Figure 2 User Interface

Company

Complaint

nametopic words

A

point

Advice topic words

4. Experiment and Evaluation
4. 1 Dataset

charge, present,

how to use, credit card

In this study, we conducted an experiment to extract the

security,

complaint and advice topic words in order to verify the feasibility of proposed method. For this experiment, we analyzed

how to save up,

Bsettinggroup friend, privacy,

the subcategory of “IT web services” of the FKC dataset,initialization, recommendation

which is under the category “industry.” We extracted 1,000

privilege, merit

negative reviews for each of the six companies. The finalCpremiumcancellation of agreement,
magazine

dataset consisted of 6,000 negative reviews in total. And weanalyzed three of the companies for experiment.
4. 2 Experiment

4. 3 Evaluation

First, we extracted the complaint topic words and deter-

We also conducted a questionnaire-based survey tomined diﬀerent threshold values for each of the three com-

evaluate the usability and eﬀectiveness of the proposed

panies. For company A, we extracted 186 complaint topic

method.The questionnaire-based survey contained following

words above the threshold value of 0.00080. For company B,

2 questions. For Q1, we extracted all nouns from the negative

we extracted 144 complaint topic words above the threshold

review for respondents to choose from. For Q2, we provided

value of 0.00076. For company C, we extracted 86 complaint

10 candidate queries for each negative review to choose from.

topic words above the threshold value 0.00080. Examples of

Q1:Please choose one word which you think could represent

the complaint topic words for each company are shown inthe dissatisfaction of the following reviews.

TABLE III. These examples show that each complaint topic

Q2:please choose the query that you think the contents re-word implies the object of diﬀerent users’ dissatisfactions.

turned by a search using this query keyword could address

the complaints found in the negative review.(multiple choicesa service and respond to diﬀerent user demands in a compre-

are allowed)

hensive way. In addition, we evaluated the eﬀectiveness and4. 4 Result and Discussion

usability of the proposed method through a questionnaire

We collected the answers of 10 respondents, and the results

survey, and the results shows thought the generated query

are shown in Table 5 and Table 6. We defined those nouns
and queries were chosen by over 5 answers as true positive.

keywords would be useful for collecting advice.
In the future, we plan to analyze other categories of the
FKC dataset. Moreover, we plan to create a website that

Table 5 Result of Q1

can automatically recommend advice web pages when users

p@1

post their complaints on the website.
Average of p@k

0.60

Acknowledgment
The result of Q1 shows that if we search by using thosenouns are with the highest value of importance for one time
only, 60% of appropriate complaint topic word can be extracted from the negative review. It not only shows that the
proposed method is eﬀective, but also explains the method
to rank nouns by calculating the importance is correct.
Table 6 Result of Q2
p@1

p@3

Average of p@k

0.70

0.50

Average of r@k

0.32

0.68

F-measure

0.44

0.58

The result of Q2 shows that if we search by using the query
which is with the lowest score for one time only, 70% of appropriate queries can be oﬀered to address the complaints
expresses in the negative reviews. What’s more, it showsthat users can acquire nearly 70% of true positive by using
query keywords those are with the lowest 3 scores insteadof 10 queries from search engine. This result does not only
demonstrate that the query keywords generated by the proposed method by scoring candidate keyword is eﬀective, butalso implies by using proposed method can help users to release their burden when searching for advice . Moreover,
it is believed that the proposed method can respond to thedemands of diﬀerent users.

5. Conclusion
In this paper, we proposed a method to generate query
keywords that will enable users to collect advice about acertain service. First, we analyzed the FKC dataset and extracted particular company names from the data. Second, we
extracted complaint topic word that can express the dissatisfaction of users from the negative reviews for each company.
Third, we extracted advice topic word that can help users tocollect advice to address their complaints. Finally, we generated a query based on the company name, complaint topic
words, and advice topic words. The advice on web pages returned by this query could address users’ dissatisfaction withIn this paper, we used FKC Data Set provided for research
purposes by National Institute of Informatics in cooperation
with Insight Tech Inc.
References[1] K. Mitsuzawa, M. Tauchi, M. Domoulin, M, Nakashima and
T. Mizumoto. “ FKC Corpus: a Japanese Corpus from New
Opinion Survey Service,” In proceedings of the Novel Incentives for Collecting Data and Annotation from People:
types, implementation, tasking requirements, workflow andresults, Portoro, Slovenia pp.11-18, May. 2016[2] T. Hasegawa and D. Kitayama. “The Visualization of Dissatisfaction Groups using Dissatisfaction Dataset,” DEIM
Forum 2017, P7-1.
[3] T. Hayashi, Y. Wang, Y. Kawai, and K. Sumiya. “An ECommerce Recommender System using Complaint Data
and Review Data,” Proc. of ACM IUI2018 Workshop on
Web Intelligence and Interaction (WII 2018).
[4] K. Utsumi, K. Inui, T. Hashimoto, H. Kimura, and
M. Ishikawa. “Extraction of Critical Knowledge concerning Social Problems and their Technological Solutions,” Socio Technology Research Journal, vol. 6, pp. 187–198, Mar.
2009.
[5] T. Sakai and K. Fujimura. “Discovering Latent Solutions
from Expressions of Dissatisfaction in Blogs,” Information
Processing Society of Japan, vol. 52, no. 12, pp. 3806–3816,
Dec. 2011.
[6] T. Hashimoto and H. Kimura and K. Inui and K. Utsumi
and M. Ishikawa. “Topic Extraction and Social Problem Detection based on Document Clustering,” Socio Technology
Research Journal, vol. 5, pp. 216-226, Mar. 2008.
[7] T. Yoshida and D. Kitayama. “An Evaluation of Feature
Term Extraction Method based on Polarity Analysis from
Customer Reviews,” IEICE-DE2016-5 vol. 116, no. 105,
pp. 19–24, Jun. 2016.
[8] H. Kakimoto, T. Hayashi, Y. Wang, Y. Kawai, andK. Sumiya. “Query Keyword Extraction from Video Caption Data based on Spatio-Temporal Features ,” Lecture
Notes in Engineering and Computer Science: Proceedings of the International MultiConference of Engineers andComputer Scientists 2018, pp. 405–408.
[9] X. Song and M. Takaku. “Study on Navigation support System based on User s Search Intents,” ARG WI2, no. 9,
2016.