Discussion Papers In Economics
And Business
The Text-Score Allocation Model: Finding Latent Topics
of Online Review Documents and Multi-Item Ratings

Sotaro Katsumata
Seungjin Kim

Discussion Paper 20-01

January 2020

Graduate School of Economics
Osaka University, Toyonaka, Osaka 560-0043, JAPANThe Text-Score Allocation Model: Finding Latent Topics ofOnline Review Documents and Multi-Item Ratings
Sotaro Katsumata∗Seungjin Kim†

Abstract
This study focuses on online review data in which comments are written in natural languages
and evaluations are attached as integers. This study develops a topic model incorporating bothnatural languages and evaluation scores, expanding latent Dirichlet allocation (LDA). The model
consists of two components: LDA and a Dirichlet-binomial clustering model. The latter assumes
binomial distributions for the review scores. Since the model assumes conjugate distributions, we
can apply a fast and stable estimator based on collapsed Gibbs sampling to estimate the parameters.
Further, the model enables us to examine the relationship between vocabulary words and review
scores based on the topic allocation results.
Keywords: Topic Modeling, Customer Reviews, Forecasting
JEL Classifications Code: M33∗ Graduate School of Economics, Osaka University, Address: 1-7 Machikaneyama, Toyonaka, Osaka 5650824, Japan,
Email: katsumata@econ.osaka-u.ac.jp, Tel: +81-6-6850-5246† Graduate School of Economics, Osaka University

1

1

Introduction

Consumer-generated media (CGM), which involves consumers transmitting and sharing information
themselves, is an important source of information for marketing activities. Kannan and Li (2017)indicate that one of the characteristics of digital marketing is that the consumers purchasing process
can be observed. This is expected to be used in consumer behavior research as a valuable source of
information, which can be obtained without conducting a questionnaire survey. (Lee and Bradlow,
2011; Humphreys and Wang, 2017).
Among CGM, online reviews that evaluate products used and purchased by the reviewers serve not
only as a source of information on consumers but also on firms. Furthermore, some studies indicate therelevance of online reviews to sales and corporate performance. The empirical study by Chevalier andMayzlin (2006) show that online book reviews aﬀect online book sales, and some studies show that online
reviews aﬀect the box oﬃce of motion pictures (Liu, 2006; Onishi and Manchanda, 2012; Dellarocas,
Zhang, and Awad, 2007). These results imply that online word-of-mouth is closely related to sales.
Additionally, this content is viewed by many potential consumers and influences their decision-making.
In addition, Luo (2009); Tirunillai and Tellis (2012) show that CGM are related to the evaluation of acompany as a whole, such as its stock price. Borah and Tellis (2016) show that CGM aﬀect the market
share of not only the focal company but also of its competitors and of companies that share its image.
These studies suggest that online reviews have a substantial impact on company performance.
However, researchers should pay attention not only to the quantity but also to the qualitative aspectsof online reviews. CGM is not a medium that companies can control. Following the three categories ofStephen and Galak (2012), the first, “paid media,” requires paying regular advertising fees; the second,“owned media,” such as company homepages owned by the company or its parent organization, can beeasily used to control the company’s message. However, the third category, “earned media,” is managedby others or by consumers and regarded as an environmental factor. Since paid media and owned media
can be controlled by companies themselves, it can be assumed that practically no negative information
about the companies or their products is conveyed through these media. On the other hand, no financial
expenditure is needed for earned media compared to paid media, and this point has been indicated as
an advantage by several studies (Berger and Milkman, 2012; Tirunillai and Tellis, 2014). However,negative reviews may be sent out by consumers (Anderson and Simester, 2014; Chen and Lurie, 2013).
Furthermore, several studies have indicated that negative reviews have a significant impact on consumer
attitudes and market outcomes (Chevalier and Mayzlin, 2006; Floyd et al., 2014; Tirunillai and Tellis,2012). Therefore, when analyzing CGM, it is important to consider the sentiment or valence of thetransmitted information, not simply the transmission volume. Heretofore, there have been studies using
dictionaries to assess these sentiments but which words express a negative impression and which words
express a positive one may vary depending on the product category. This limits the analysis that canbe performed using a general dictionary.
Therefore, the rating attached to a review is important. Ratings are often given as discrete integerscores, and considering these ratings and the words that appear in the review text, one can obtain
information that will greatly contribute to marketing research and product development. However, few
models exist that can be easily used to quantify a natural language such as the text of a review text
and to analyze its relationship with an integer score such as a review score. Therefore, in this study,
we present a classification model that considers both online review ratings and text.

2

Literature Review

2.1

Natural Language Analysis

Research on how to utilize information written in natural language, especially that posted by consumers,has generated many studies and is one of the most popular issues in marketing research (Berger et al.,2019; Humphreys and Wang, 2017; Wedel and Kannan, 2016). As already mentioned, online review
analysis requires not only counting how many posts there are but also examining their valence and2sentiments, so it is necessary to not only perform a simple aggregation but also to analyze posts’
contents.
Since CGM posts are written in natural language, their contents need to be quantified by an appropriate method. Text data are called Unstructured data (Balducci and Marinova, 2018). Recently,
Balducci and Marinova (2018) have oﬀered several quantification methods based on previous studies.
Furthermore, Berger et al. (2019) present a procedure for analyzing text data. They show three tools forclassifying text data. The first tool is for estimating moods and sentiments from the words themselves.
For example, the sentiment expressed by a word can be determined by LIWC (Linguistic Inquiry andWord Count), which is used in Berger and Milkman (2012) and Hewett et al. (2016). The polarity(sentiment) dictionary makes it possible to separate negative reviews from positive ones. The second
tool is for classifying words and extracting topics. By exploring and classifying topics that do nothave external criteria, it is possible to interpret the object and obtain information. The third tool isfor examining the relationship between words in a sentence. In the present study, a model for topic
extraction is constructed and used for the analysis. In the next section, we will review the research on
latent Dirichlet allocation (LDA), a representative topic model.

2.2

Expansion of the LDA

LDA and its extended model are widely used in marketing as a method for classifying sentences. LDA
is a model proposed by Blei, Ng, and Jordan (2003), which assumes a high-dimensional categorical distribution for the words appearing in sentences and assumes a Dirichlet distribution that is its conjugate
distribution as a prior distribution. In machine learning, there are many models that are extensions of
LDA. For example, the author-topic model (Rosen-Zvi et al., 2004) considers author factors, the dynamic topic Model (Blei and Laﬀerty, 2006) considers time series factors, and the Pachinko allocation
model (Li and MaCallum, 2006) incorporates hierarchical classification parameters. In addition, LDA
can use collapsed Gibbs sampling, which is a stable and fast estimator (Griﬃths and Steyvers, 2004).
Collapsed Gibbs sampling is a kind of Markov-Chain Monte Carlo (MCMC) estimator.
In the marketing literature, Liu, Burns, and Hou (2017) present an analysis model for SNS text andapply it to Twitter data. In addition, Tirunillai and Tellis (2014) apply an extended LDA model toonline reviews, and Büschken and Allenby (2016) present the results of an extension of the author-topic
model. The author-topic model is also based on the model of Nam, Joshi, and Kannan (2017). Further,Toubia et al. (2019) apply an extended topic model to CGM. LDA itself can be applied to any data
which can be assumed to follow a multinomial distribution and is also applicable to areas other thannatural language. Trusov, Ma, and Jamal (2016) have developed a model that expands the correlated
topic model (Blei and Laﬀerty, 2007) and conducts an empirical analysis of website visit behavior.
Using collapsed Gibbs sampling as an estimator, the parameters of an LDA model can be obtained
from the conditional posterior distribution in the same way as in other Bayesian models. This implies
that LDA can be easily expanded and combined with other models and its parameters can be estimatedby the Metropolis-Hastings (M-H) method. Trusov, Ma, and Jamal (2016) and Büschken and Allenby(2016) use the M-H method as part of the parameter estimation. However, since the M-H method has
a high computational load and it is diﬃcult to stabilize the estimation result, ideally, it is desirable tocreate a model using only combinations of conjugate distributions. Therefore, in this study, we present
a model that assumes a binomial distribution, taking advantage of the property that a review score isa continuous integer with a maximum value.

3

Model

3.1

Settings and Notation

In this section, we define the model. In the first part of this section, we define LDA, and then the score
allocation model (SAM), which assumes a binomial distribution for the input data. Subsequently, thetext-score allocation model (TSAM), our proposed model, is defined by combining these two models.

3

Assume that there are D pairs of review comments and ratings. We call each review a document.
Document d includes both text and score data. The text is decomposed into words. Let the total
number of words be N ; wi is the i-th（i = 1, · · · , N ） observed word. There are V vocabulary words
observed in the data. We assume that for the i-th word, any one of the V vocabulary words is actuallyobserved; therefore, wi is a V -dimensional vector, of which each element takes the value 1 if thecorresponding vocabulary word is observed and the value 0 otherwise. Therefore, wiv = 1 if vocabulary
word v(v = 1, · · · , V ) is observed and then wiv′ = 0, ∀v ′ ̸= v. Note that each of the N words may
belong to any one of the D documents. We introduce word-document indicator xi , a D-dimensional
vector, as follows: if the i-th word is observed in document d, xid = 1; otherwise, xid = 0.
Next, we define the notation for ratings. Assume that there are J item ratings in review d and theobserved scores are denoted by ydj . For item j, a score is given by reviewers on a Qj + 1-point scale.
Let the lowest score be 0, so ydj = {0, 1, · · · , Qj }.

3.2

Text Allocation Model: Latent Dirichlet Allocation (LDA)LDA was proposed by Blei, Ng, and Jordan (2003) and assumes that word wi follows a V -dimensional
categorical distribution (a single-trial multinomial distribution):( )wi ∼ CategoricalV ϕ̃i(1)which has K latent parameters ϕ̃i called topics. Let us introduce the K-dimensional parameter zi . If
wi belongs to topic k, zik = 1 and zik′ = 0, ∀k ̸= k ′ . The elements of the latent parameters are defined
as follows:ϕ̃iv =

K∏

ϕzkvik(2)

k=1

In addition, assume that zik follows a categorical distribution where xi is the word-document indicator defined above.
D
( )
∏
xid
zi ∼ CategoricalK θ̃i , θ̃ik =
θdk
(3)
d=1

As prior distributions, ϕk and θd follow Dirichlet distributions that are conjugates of the categorical
distribution in equation (1). Let ϕk ∼ Dirichlet(β) and θd ∼ Dirichlet(α); the full conditional posterior
distribution is obtained as follows:
π(z, Φ, Θ|w) ∝ π(w|z, Φ)π(z|Θ)π(Φ)π(Θ)(4)

We can obtain posterior sample of the unknown parameter z by using the collapsed Gibbs sampling
method proposed by Griﬃths and Steyvers (2004).
∫ ∫π(z|w) ∝π(z, Φ, Θ|w)dΦdΘ
(5)
The conditional posterior distribution of zi is easily obtained as follows:π(zik |z−ik , w) ∝nkv∗ ,−i + β(nd∗ k + α)
nk,−i + V β(6)∗where v ∗ is observed vocabulary at i-th word and,the document the i-th
∑Ntherefore, xiv∗ = 1. d is∑
N
′ ∗
′
∗
word belongs to; therefore, xid∗ = 1. nkv∗ =i′ =1,i′ ̸=i xi v zi k , nk,−i =v=1 nkv , and nd k =
∑N
′
∗
x
z
.
GriﬃthsandSteyvers(2004)
have
shown
that
collapsed
Gibbs
sampling
isafast,
i′ =1,i′ ̸=i i d dk
stable estimator and steeply converges to the stationary distribution.

4

3.3

Score Allocation Model (SAM)

LDA only focuses on the classification of natural language. On the other hand, the SAM defined in thissection is a model for the classification of multi-dimensional scores observed in a review. First, let us
assume that the rating of item j in review d follows a Binominal distribution:
ydj ∼ Binomial(Qj , ψ̃dj )(7)where we assume the same K latent topics for ψ̃dj as in LDA. Let the topic indicator parameter be cd ,which is a K-dimensional vector. If document d belongs to topic k, cdk = 1, and cdk′ = 0 for k ′ ̸= k.
Then,
K
∏
cdk
ψ̃dj =
ψkj(8)
k=1

Similar to the LDA, we assume the following categorical distribution for cd :cd ∼ CategoricalK (θd )(9)For ψkj , we assume a Beta distribution, a conjugate of the binomial distribution. For cd , we assume
a Dirichlet distribution. Let ψkj ∼ Beta(γ) and θd ∼ Dirichlet(α); the full conditional posterior
distribution is obtained as follows:π(c, Ψ, Θ|y) ∝ π(y|c, Ψ)π(c|Θ)π(Ψ)π(Θ)(10)

Since this model consists of the conjugate distributions, the conditional posterior samples of cd are
generated by collapsed Gibbs sampling.
∫ ∫π(c|y) ∝π(c, Ψ, Θ|y)dΨdΘ
(11)
The conditional posterior samples for cd are generated from the following categorical distribution:π(cdk |c−dk , y) ∝

J
∏
j=1

∏ydj

∏Qj −ydj(nkj0,−d + q + γ)
q=0 (nkj1,−d + q + γ)
q=0∏Q j
q=0 (nkj,−d + q + 2γ)

(12)

∑D
∑D
∑D
′
′
′
′
where nkj1,−d =d′ =1,d′ ̸=d yd j cd k , nkj0,−d =d′ =1,d′ ̸=d (Qj − yd j )cd k , and nkj,−d =d′ =1,d′ ̸=d
Qj cd′ k .
If cd is given, parameter ψkj can be easily obtained:ψkj =nkj1 + γ
nkj + 2γ

(13)

∑D
∑D
where nkj1 = d=1 ydj cdk and nkj = d=1 Qj cdk . ψkj is a useful parameter for examining the scoreof item j when document d belongs to topic k. Since the expected value of the score is Qj ψkj , a largerψkj implies a better rating. The model is regarded as a clustering technique and ψk is regarded as themean vector of cluster k. In the appendix, we compare the results of the SAM with those of k-means
using sample data. In addition, if K < J, a K × J matrix Ψ = {ψkj } can be interpreted as a dimension
reduction matrix from the K-th to the J-th dimension.

3.4

Text-Score Allocation Model (TSAM)

By combining LDA and the SAM, we define the TSAM, which assumes a common topic. Mimno et al.
(2009) have proposed the polylingual topic model (also known as the joint topic model), which merges
two text data sets sharing a common topic. However, this study also involves two distinct data sets:
reviews and scores.

5

The model has two input data sets. First, we assume that the score of review d follows the binomial
distribution. At this time, the parameter depends on a latent topic, and the parameter cdk indicatesthe aﬃliations for topic k.
ydj ∼ Binomial(Qj , ψ̃dj ), ψ̃dj =

K
∏

cdk
ψkj

(14)

k=1

Second, as in LDA, the observed word wi follows the categorical distribution below.
K( )∏wi ∼ CategoricalV ϕ̃i , ϕ̃iv =ϕzkvik(15)

k=1

The model has two latent parameters: zi and cd . To connect these two latent parameters, we assume
there exists a common prior parameter θd . If word wi is aﬃliated with document d, the latent parameterzi follows a categorical distribution with parameter θd . Further, cd follows a categorical distribution
with parameter θd :
D
( )
∏
xid
zi ∼ CategoricalK θ̃i , θ̃ik =
θdk

(16)

d=1

cd ∼ CategoricalK (θd )(17)

For prior parameters, we assume a Dirichlet distribution for ϕi and θd ; for ψkj , we assume a Beta
distribution, which is the conjugate of the binomial distribution. Therefore, Φi ∼ Dirichlet(β), θd ∼Dirichlet(α), and ψkj ∼ Beta(γ). The full conditional posterior distribution is as follows:
π(z, c, Φ, Ψ, Θ|w, y) ∝ π(w|z, Φ)π(z|Θ)π(Φ)π(y|c, Ψ)π(c|Θ)π(Ψ)π(Θ)(18)

As in LDA and the SAM defined above, this model also has an analytical solution for implementing
collapsed Gibbs sampling.
∫ ∫ ∫π(z, c|w, y) ∝π(z, c, Φ, Ψ, Θ|w, y)dΦdΨdΘ
(19)
Therefore, we need to estimate only two parameters, z and c, whose posterior samples are obtained asfollows:
nkv∗ ,−i + β(nd∗ k + cd∗ k + α)(20)π(zik |z−ik , c, w) ∝nk,−i + V β
where nkv∗ ,−i , nk,−i , and nd∗ k are the same those defined above for LDA. A posterior sample for cdk is
obtained from the following distribution:∏Qj −ydj
J ∏ydj
∏
(n
+
q
+
γ)
(n
+
q
+
γ)
kj0,−dq=0 kj1,−dq=0
 (ndk + α)π(cdk |c−dk , z, y) ∝ 
(21)
∏Qj
(n
+
q
+
2γ)
q=0 kj,−dj=1where nkj1,−d , nkj0,−d , and nkj,−d are the same as those defined for the SAM above.
If z and c are given, the prior parameters ϕkv , ψkj , and θdk can be obtained as follows:
ϕkv =nkv + βnk + V β(22)

nkj1 + γ(23)nkj + 2γ
ndk + cdk + α
(24)
θdk =
nd + 1 + Kα
Figure 1 shows a graphical model of the three models above. TSAM is obtained by combining LDA
and the SAM. The parameter θ is shared, and the potential topic allocation parameters c and z are
generated by using this parameter.
ψkj =

6

Text Allocation Model (Latent Dirichlet Allocation LDA)
Text-Score Allocation Model (TSAM)

α

θ

z

β

w
Nd D

γ

ψ

V

c

y

z

w

φ

Score Allocation Model (SAM)

α

θ

c

α

Nd D

y
D

γ

J

θ

β

φ
V

ψ
J

Figure 1: Graphical Expression of Three Models

3.4.1

Forecasting

Since, in the estimation, the samples are obtained from the conditional distribution by Gibbs sampling,
predicted values can be obtained by using an algorithm for missing value augmentation. For documentd, if the natural language w of a review is observed but the evaluation score ydj is missing, the model
can generate a predicted value for it from the following distribution:
(
)(
)q (
)Qj −qnk∗ j1,−d + γ
nk∗ j0,−d + γ
Qj
π(ydj = q|y−dj , c, z) =(25)
qnk∗ j,−d + 2γ
nk∗ j,−d + 2γ
where k ∗ is the topic document d belongs to; therefore, cdk∗ = 1.
One of the simplest ways to generate the random samples of ydj following the distribution above, isrepeated Bernoulli trials. Generate r(h) from r(h) ∼ Bernoulli ((nk∗ j1,−d + γ)/(nk∗ j,−d + 2γ)) Qj times∑Qj (h)and summarize the results. Then, ydj = h=1r
is a random sample from the binomial distribution
above.

4

Data

4.1

Data Collection

The data are reviews of the Japanese price comparison site Kakaku.com (https://kakaku.com). Kakaku.com
is a website operated by Kakaku.com, Inc., where visitors can view comparisons of actual sales pricesof electrical appliances and other product/service categories, such as alcoholic beverages and types of
insurance, and consumer reviews of the products/services. Reviewed and evaluated products/services
are available mainly in Japan and reviews are mostly written in Japanese. However, according to SimilarWeb (https://www.similarweb.com), Kakaku.com is the most visited website in the “E-commerceand shopping: Price Comparison” category in the world.

7

160

In this study, we select a smartphone as the objective product category. In particular, we focus on
the iPhone series, which are manufactured and sold by Apple, Inc. The analysis period is nine and ahalf years: from August 2008, when iPhone 3G was released in Japan, to February 2018. During thistime, 13 models have been released, and reviews of these models are in the objective data set. For datacollection, we use Python to scrape websites. The data include the posting date and time of the review,the name of the product (model) to be reviewed, the review title, the review text, the reviewer, and thescores. The scores comprise six items—the five part-wise evaluations and the overall satisfaction—and
all are scored on a 5-point scale. Among these, satisfaction is a comprehensive evaluation, and theother five aspects are partial evaluations. Specifically, they are design, mobility, response, screen, andbattery. For example, the score for design is the specific evaluation of a model’s appearance. This
includes evaluation of colors. The score for mobility evaluates a model’s ease of portability as a mobile
device. The score for screen evaluates the size and resolution of the screen. Finally, with regard tobattery, the evaluated element is battery durability. In the following, scores for the five aspects otherthan satisfaction will be collectively called subscores.
Of the collected reviews, 5,375 reviews were finally included in the analysis, excluding those that
did not have a full text or that lacked scores for the 6 items. Fig. 2 shows the total number of reviews
used in the analysis, by month, and provides the release month of the models. As can be seen, thenumber of reviews has increased greatly in the months when new models were released. It can also beseen that reviews have been observed throughout the entire period.

140●

iPhone 6●

iPhone 5
iPhone 4S

●

●

120

●

●

iPhone X

iPhone 3G

100

●

●

iPhone 7
●
●

●

●

●

●

●

●

●●●

●

●

●
●●

iPhone 8

●

●

●●

●
iPhone SE
●

●●

●

●
●

●●

●

●

●

●
●
●

●

●

201712

201012

201006200912

200906●●●201706●

iPhone 6s
●

●

●

●

●

200812

●●

●

●

●

●
●

●
●

201612

●

●

●●

●

●

201206

20

●

●●

●

201606●●●201506

●

●

●

●
●

201412

●

●

●●

●

201406●●●●201306●●201112●●

201212

●

●●

●

●
●

201106

●

●

●

●

●

●

iPhone ●4
iPhone 3GS●

201512●

20131280●

60

number of reviews●●
iPhone 5s
●

40

●

●

●

Figure 2: Number of Reviews and Release Date

4.2

Data Cleaning

Since the review text to be analyzed is written in Japanese, a Japanese morphological analyzer needs to
be used. In this study, words are decomposed using MeCab, a general-purpose morphological analysis
engine developed by Kudo, Yamamoto, and Matsumoto (2004). In addition, ipadic-Neologd, a newneologism dictionary developed and managed by Sato, Hashimoto, and Okumura (2017), is used as adictionary to decompose words. Nouns, verbs, and adjectives are extracted and used in the analysis.
In review text, a negation tag may be used for verbs and adjectives. In Japanese sentences, anauxiliary verb for negation is added after the verb or adjective. Therefore, we check whether such an
auxiliary verb appears after a verb or adjective is observed, and if it does, a negation tag is attached tothe target verb or adjective. As a result, in the decomposition, a verb or an adjective with a meaning
opposite to that of its regular form is counted as a separate word by adding (-1) after the word. For

8example, if the word Ugoka-nai (does not move) is observed, with the negation expression nai (not)attached to the verb Ugoku(move), the word move (-1) is recorded in the data set.
Next, we need to clean the decomposed words. Cleaning consists of the following four steps: (1)exclude words observed only one time, (2) exclude words with length 1 (single-character words), (3)exclude words with only numerals, (4) exclude the product name (iPhone) and words related to themodel name (e.g., 3G), and (5) exclude general indicating pronouns (e.g., this, that). As a result of thecleaning, 567,249 words composed of 12, 815 vocabulary words were obtained. There are 5,375 reviews,
the shortest of which contains 5 words and the longest of which contains 506 words.

4.3

Distribution of Scores

The number of documents D = 5375, the number of evaluation items J = 6, the maximum score foreach evaluation item is Q − 1 = Qj − 1 = 4, ∀j = 1, · · · , J, the number of vocabulary words V = 12, 815,
and the corpus length N = 567, 249. The evaluation data y consist of a D × J matrix that takes
0, · · · , Q − 1 values, and the natural language w is an N × V matrix that takes binomial values {0, 1}
for each element. Further, the document index x is a N × D matrix that takes the binomial values{0, 1} .
The observed score distribution is shown in 1. This table shows that all the items exhibit highaverage scores. For design and screen, over 70% of reviewers give the highest rating of 4, whereas very
few reviewers give a rating of 0 or 1. For satisfaction, which is the comprehensive evaluation 60% ofreviewers also give a rating of 4. This implies that the consumers reviewing the products tend to give arating of 4 if they are generally satisfied, and a rating of 3 or less if the product has any clear defects.
Table 1: Summary of Scores
Score
0
1
2
3
4
Total

4.4

Design
59
1.1%
78
1.5%
344
6.4%
1080
20.1%
3814
71.0%
5375
100.0%

Mobility
101
1.9%
232
4.3%
687
12.8%
1485
27.6%
2870
53.4%
5375
100.0%

Subscores
Response
62
1.2%
128
2.4%
405
7.5%
1091
20.3%
3689
68.6%
5375
100.0%

Satisfaction
Screen
44
0.8%
52
1.0%
329
6.1%
988
18.4%
3962
73.7%
5375
100.0%

Battery
216
4.0%
494
9.2%
1212
22.5%
1825
34.0%
1628
30.3%
5375
100.0%

159
185
374
1458
3199
5375

3.0%
3.4%
7.0%
27.1%
59.5%
100.0%

Forecasting and Model Comparison

To assess the forecasting performance of the model, some observed scores are masked. Hereafter, thereviews in which none of the information is masked are called complete reviews, and the prediction
samples are not based on these reviews. We prepare two types of samples for prediction. The first type
involves a subset of the reviews where text data and score data other than satisfaction are used; only
the satisfaction score is not used. In this subset, satisfaction has actually been evaluated on a scale of
0 to 4, but these data are masked. On the other hand, the review text and the subscores of the fiveaspects are not masked. This set of reviews is called text-subscore reviews. In text-subscore reviews,
the satisfaction scores are predicted by the text and subscores. Next, we prepare text-only reviews,
from which lack satisfaction and subscores are omitted. We divide newer posts, those submitted after
2015, into three groups to prepare text-subscore reviews and text-only reviews. One-third of each is notmasked and classified as complete reviews. We masked the satisfaction ratings of another one-third of
the posts to prepare text-subscore reviews and masked the satisfactions and other five subscore ratingsof the remaining one-third of the posts for text-only reviews. The size of each set of reviews is shownin Table 2, where NA implies the masked scores. In model estimation, all the observations in each set
are included in the analysis and masked scores are augmented by MCMC simulations.
The model presented in this study needs the number of potential topics to be provided, as does
LDA. Therefore, to determine the most appropriate number of topics, some models that assume different numbers of topics are used, to compare their fit and predictive performances. In this study, a9Table 2: Summary of Three Datasets
Text
Subscore (5 aspects)
Satisfaction# of reviews
2008-2014
2015
2016
2017
2018
Total
% of reviews

Complete reviews
available
available
available

Text-subscore reviews
available
available
NA

Text only reviews
available
NA
NA

3849
138
171
174
25
4357
81.1%

0
138
171
174
26
509
9.5%

0
138
171
174
26
509
9.5%

total of 15 models where the number of topics is K = 2, 5, 10, 20, · · · , 100, 110, 120, 150, 200, 300 are
estimated. Since the model is estimated by the MCMC method, it is necessary to determine the numberof iterations. According to Griﬃths and Steyvers (2004), an LDA estimated by collapsed Gibbs sampling converges to the posterior distribution within 1,000 iterations; therefore, the first 1,000 iterations
are withdrawn as burn-in and we collect the samples of subsequent 2,000 iterations to determine theposterior density. The parameters included in the model are z and c. In addition to this, the missing
score data y also need to be estimated. The satisfaction scores missing from the text-subscore reviews
and the subscores and satisfaction scores missing from the text-only dataset are obtained from theposterior distributions. In the simulation, samples for all parameters are generated sequentially basedon the other samples.
To compare the models’ forecasting accuracy, hit rate, AUC (Area Under Curve)—also known as
ROC (Receiver Operating Characteristics)—score, and prediction MSE (Mean Squared Error) are used.
For the hit rate, the expected value of the predicted score is rounded to an integer to obtain the predictedvalue. Since scores are five ordered integers, we consider a predicted value up to ±1 of the actual value
to be a correct prediction. Therefore, our hit rate is not a strict evaluation but a hit rate with nearest
neighbors, which is also applied in Ansari, Essegaier, and Kohli (2000). Although the AUC score is only
applied in binomial discrimination problems, the binary value of {satisf action = 4, satisf action ≤ 3}
is used as the prediction target. For the prediction MSE, the expected values of the scores are comparedwith the observed integer.
To compare the prediction accuracy of our model with that of other models, we estimate some
regression models. Specifically, an ordered logit model of 5-point dependent variables is estimatedto compare the hit rate (with nearest neighbors) and MSE, and the binomial logit model ({1, 0} ={satisf action = 4, satisf action ≤ 3}) is estimated to compare AUC scores. In the predictions for textscore and text-only reviews, a set of complete reviews is used to obtain the parameters. To forecast
text-subscore review scores, the 5 subscores and the number of observations for the top 50 words (mostfrequently observed in the set of text-subscore reviews) are used as explanatory variables. Therefore,
the number of explanatory variables in the model is 5 + 50 = 55. To forecast text-only review scores,
the number of observations for the top 50 words is also used as an explanatory variable. However, themodel does not include the subscores; therefore, the number of explanatory variables is 50.

5

Results

5.1

Model Comparison

As defined above, several indices are calculated to assess model fit. First, the log of the marginal
likelihood is calculated as an index for comparing the in-sample fit. The marginal likelihood is obtained
from the harmonic mean, which is also used in Griﬃths and Steyvers (2004) to determine the optimal

10

number of topics.
The upper left part of Figure 3 shows the relationship between the number of topics and the models marginal likelihood. The marginal likelihood is an index whose larger values are assumed to be better.
From here, it can be seen that the fit increases rapidly up to approximately K = 80, it is largest
when K = 110, and it gradually decreases for larger values of K. Therefore, based on the marginal
likelihood, it can be said that K = 110 corresponds to the most appropriate model. However, themarginal likelihood shows the in-sample fit of the model. Therefore, we also examine the accuracy
using other indicators.

●

●

● ●

● ● ● ● ● ●Forecasting Hit Rate
1.00

−3000000

Marginal Likelihood

●

●

0.95

●●● ●
●●

●

● ●

● ● ● ● ● ●

●

●

●

●

●

●

● ● ● ●

● ● ●

● ●

0.90

●

0.80

●

100

150

200

250

300

0

50

100

150

200

K

K

Forecasting ROC

Forecasting MSE

250

300

0.7

●
●

●
●
●

0.6

●

●

● ●
●

● ● ● ●

●
●

1.0
0.7

●
●

●
●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

●

●

●

0.9

●

0.8

●

MSE (Mean Squared Error)● ● ● ● ● ● ● ●0.8●●● ● ● ● ● ● ●

1.1

1.2

50

0.9

0

AUC (Area Under Curve)● ● ● ● ● ●0.85−3400000●●●−3800000

log marginal likelihood●

Hit Rate (with Neighbors)

●

●

●

●
● ● ● ● ● ●● ● ● ● ● ● ● ● ●

0.6

0.5

●

0

50

100

150

200

250

300

0

50

100

K

150

200

250

300

K

Upper left: log marginal likelihood. Upper right: Hit rate (with the nearest neighbors). Bottom left: AUC.
Bottom right: Forecasting MSE. The solid line is the prediction results for the text-subscore reviews, and thedotted line those for the text-only reviews. The prediction target is the satisfaction scores. Hit rate and MSE
indicate the prediction results in terms of the 5 ordered integers, and AUC indicates the prediction resultsof a binomial variables for which {1, 0} = {satisf action = 4, satisf action ≤ 3}. The straight lines are theprediction results of the regression models (the ordered logit model and the binomial logit model).

Figure 3: Results of Forecasting

The upper right, lower left, and lower right of Figure 3 are the prediction results for the text-subscorereviews and text-only reviews. The prediction target is satisfaction score. The upper right part of the
figure is the hit rate including the nearest neighbors, the lower left is the AUC score calculated from the
ROC curve, and the lower right is the forecast MSE. The hit rate is determined for the five values as theratio of the diagonal component ±1 elements of the predicted/observed cross-tabulation. AUC predicts
the binomial result of whether a score is 4 or not. The larger the hit rate and ROC are, the better. On
the other hand, the smaller the MSE is, the better. In the figure, the solid line is the prediction results

11

for the text-subscore reviews, and the dotted line is the prediction results for the text-only reviews. The
straight lines are the prediction results of the regression models the ordered logit model and binomial
logit model), where the solid lines are the predictions for the text-subscore reviews, and the dotted lineis the prediction for the text-subscore reviews.
Theoretically, AUC is expected to be 0.5 when predicted completely at random. Therefore, theprediction ability of all models is better than random prediction. In addition, compared with theresults of the regression models, the results show that the proposed TSAMs with diﬀerent numbers of
topics have a generally better predictive ability. The regression models performed better than some
TSAMs compared with small K. However, when K increases to some extent, the TSAM’s accuracy
is higher than that of the regression models, with a few exceptions (e.g., AUC for text-only reviews
for the K = 150 model). In addition, the diﬀerences between the TSAM’s results and the regression
models’ results for the text-only reviews are generally larger than for the text-subscore reviews. Thisis because the regression models include only the top 50 words as explanatory variables to forecast
text-only review scores, whereas the TSAM classifies topics based on more than 10,000 words. Thisimplies that the predictive accuracy increases by utilizing this massive number of words.
For each part of the figure and for all models, the prediction results for the text-subscore reviews aremuch better than the prediction results for the text-only reviews. This implies that the relationships
between the five subscores and satisfaction are strong. However, even if the number of topics increases,
the prediction ability does not greatly improve.
From the model comparisons, the model with the highest marginal likelihood was for K = 110, but
it can be seen that there is almost no change in the marginal likelihood from K = 100 to K = 130.
Regarding the hit rate, AUC, and MSE, TSAMs with small values of K do not exhibit good prediction
accuracy. However, the figure shows that the predictive accuracy increases until K is approximately 20
to 30. After that, the predictive accuracy does not dramatically improve. Therefore, in the following,
we examine the result of the model with K = 110. The sampling path of this model is shown in Figure
7 in the Appendix. It can be seen that this path converges to the posterior distribution in fewer than
1,000 iterations.

5.2

Topic Interpretation

In this section, we examine the results of the topic analysis obtained from the model. In an ordinary
topic model such as LDA, researchers have to interpret the characteristics of topics based on the words
that have high generating probabilities for each topic. In addition, researchers must also examine
whether a topic is a good topic or a bad one. However, our TSAM incorporates review scores andwe can use the expected scores as a guideline. Parameter ψkj , which aﬀects the expected value of areview score, is estimated for each topic. By examining this ψkj , we can intuitively interpret whether
the topic is a compilation of good or bad reviews. Based on these parameters, it is possible to assess
whether a word with a high probability of belonging to each topic is related to a good review or abad one. Furthermore, since the evaluation is divided into five subscores in addition to the satisfaction
score, which is an overall evaluation, it is possible to examine how highly each of the five aspects wasevaluated.
Table 3 shows the topics for which the value of ψkj in any of the six scores is in the top three amongthe 110 topics. Conversely, Table 4 shows the topics for which the value of ψkj in any of the six items
is in the bottom three places among all the 110 topics. Topics for which any of the six scores are in
the top three are called good topics, and topics for which any of the six scores are in the bottom three
are called bad topics. The numbers shown in the table are the posterior expectations of ψkj , and thenumbers in parentheses indicate the ranking across all topics. In addition, the top 10 words obtained
based on ϕkv are shown in the right column. Note that, since the model analyzed Japanese words andthe words in the tables are translated, some words consist of more than two English words.
First, from Table 3, we can see that positive words appear frequently in the top words. For example,
in Topic No. 73, there are many words that can be interpreted as corresponding to a good rating, suchas highest and excellent. For topics No. 73, No. 65, and No. 2, the most relevant words are highest, love,and great, respectively, and the topic with the highest rating is a positive word. However, it is diﬃcult

12

to understand exactly what aspects are related to good evaluations by using only words with a positive
meaning. We cannot examine which aspect is excellent or great without subscores. Our model allows
us to easily examine the relationship between scores and frequent words. For example, in topic No. 21,
words such as thin, light, and fast are frequently observed. This implies that a thin and lightweight
shape and high-speed processing lead to a positive evaluation of mobility and response, which rank
third among the 110 topics. These results indicate that some words are closely related to the subscores
given by the reviewers and analysts can interpret these relationships to find the characteristics of each
topic.
On the other hand, there are topics where high and low ratings are greatly divided among items.
No. 53 in Table 3 exhibits very high ratings for design and screen, whereas its ratings for mobility
and battery are in the bottom 10% of all topics. This suggests that a certain number of consumers
were satisfied with the design and appearance of the product but experienced issues with battery life.
Regarding No. 71, battery has a high rating, whereas design and mobility have low ratings. We findthat words such as large and huge are frequently observed in the reviews of topic No. 71. These results
suggest that the words big and huge were observed when reviewers were evaluating the diﬃculty ofholding the product and are related to the lower scores in design and mobility.
No.
73

Design0.981(2)

Mobility Response Screen
0.959
0.98
0.984
(2)
(2)
(2)

Battery
0.859
(1)

65

0.976
(3)

0.926
(8)

0.973
(6)

0.979
(5)

0.825
(6)

2

0.968
(11)

0.928
(7)

0.973
(8)

0.98
(4)

0.849
(2)

21

0.95
(34)

0.948
(3)

0.978
(3)

0.977
(7)

0.793
(14)

50

0.974
(4)

0.975
(1)

0.985
(1)

0.982
(3)

0.796
(11)

53

0.985
(1)

0.729
(100)

0.813
(95)

0.988
(1)

0.429
(105)

71

0.763
(106)

0.552
(106)

0.942
(30)

0.959
(27)

0.846
(3)

Satisfaction Frequent words
0.971
highest, excellent, response, design, have,(1)
mobile, relinquish (-1), luxury, incomparable,
all
0.966
satisfaction, change-model, very, love, over(2)all, concern, comprehensive, demerit-point,change-device, long-lasting
0.964
great, have, single-word, all, excitement, de(3)sign, perfect, only-this, mistake (-1), suﬃcient
0.956
thin, light, fast, rather, map, long, speed, sur(4)prise, vertical, lightness
0.954
LTE, area, communication, tethering, cor(6)respondence, my-home, internet-speed, network, speed, Mbps
0.947
mobile, internet, PC, able, email, see, pleas(11)ant, site, review, convenient
0.85
large, screen, see, conspicuous, huge, round(77)ness, have, design, thin, take-on

Note a) The numbers shown in the table are the posterior expectations of ψkj , and the numbers in parentheses
indicate the ranking.
Note b) Since Japanese words were analyzed in the model and the words in the tables are their translation,some of them consist of more than one English word.
Note c) The sign (-1) added after some words indicates negation.

Table 3: Good Topics

Next, Table 4 shows topics with low ratings. Contrary to topics with high ratings, the words bad,worst and disappointment are found here. However, looking at the evaluations by the five aspects,
there are topics where high evaluations and low evaluations are mixed. The most remarkable topic is
No. 49, which shows that the overall satisfaction rating is very low, at the 109th place, even thoughmobility and screen ratings are ranked high, within the top 10%. Looking at the evaluations by thefive aspects, response and battery ratings are also low, but the reason why satisfaction is lower thanthat can be understood by looking at the frequently observed words. Looking at the right column ofTable 4, the most frequently occurring words are related to issues with telecommunication qualities;
examples of these words include: signal, out-of-service, antenna, and weak, and SB (abbreviation for
the telecommunications company SoftBank). In addition, carrier also indicates a communications
company. These words imply that these low evaluations are not for the product model itself but for thecommunications environment. Issues with telecommunication qualities are strongly related to the low
degree of satisfaction. Although this review site itself is not for a telecommunications company but an

13online community for evaluating mobile terminal models, consumers often confuse the quality of serviceof the models provided by manufacturers with the quality of the communications environment provided
by telecommunications companies. Therefore, since an evaluation as low as the one mentioned above
depends on the communications environment, it is not necessary for the manufacturer to improve its
own product in response. It can be said that, instead, it is the telecommunications companies that haveto improve the environment for comfortable communication services.
In the following sections, we will further examine each topic. The next section focuses on iPhone
generations and examines topics that frequently appear in each generation.
No.
85

Design0.402(110)

Mobility Response Screen
0.412
0.624
0.55
(109)
(105)
(109)

Battery
0.332
(108)

49

0.95
(35)

0.921
(9)

0.871
(85)

0.976
(8)

0.562
(97)

47

0.518
(109)

0.396
(110)

0.491
(109)

0.529
(110)

0.333
(107)

39

0.764
(105)

0.682
(103)

0.617
(107)

0.614
(108)

0.492
(104)

93

0.896
(83)

0.655
(104)

0.403
(110)

0.832
(103)

0.267
(109)

41

0.836
(99)

0.424
(108)

0.612
(108)

0.859
(99)

0.254
(110)

9

0.609
(108)

0.505
(107)

0.881
(80)

0.892
(85)

0.728
(50)

Satisfaction Frequent words
0.252
bad, extremely, able (-1), disappointment,(110)
usable (-1), not-good, worst, can (-1), lessthan, slow
0.276
signal, telephone-call, out-of-service, place,(109)
situation, SB, quality, carrier, antenna, weak
0.344
screen,operation,
conversion,bright,(108)
telephone-call, sound-quality, hard-to-do,
letter, menu, display
0.406
exchange, correspondence, get, model-body,(107)
failure, support, status, repaire, -able, malfunction
0.504
email, receive, incoming, -able, able (-1), con(106)firmation, setting, correspondence, MMS, notification
0.616
phone, mobile-phone, function, mobile, re(105)view, touch-panel, usual, dial, pleasant, necessary
0.705
button, home, push, power-supply, volume,(100)
screen, location, arrangement, hard-to-do,
sleep

Note a) The numbers shown in the table are the posterior expectations of ψkj , and the numbers in parentheses
indicate the ranking.
Note b) Since Japanese words were analyzed in the model and the words in the tables are their translation,some of them consist of more than one English word.
Note c) The sign (-1) added after some words indicates negation.

Table 4: Bad Topics

5.3

Generations and Topics

We focus on multiple generations of the iPhone series models released between 2008 and 2018. We can
examine the relationship between model reviews, topics, words, and ratings. Assume that there are D
reviews, and let the set of reviews written for generation g be denoted by Dg and the number of reviews
included in the set be n(Dg ). The topic share (appearance probability of topic k) for each generation
g is obtained from the average value of the appearance probability the topic in each document θ̄dk .
θ̄gk =∑1θdkn(Dg )(26)d∈Dg

Table 5 summarizes the top three topics, having the largest values of θ̄g for all generations. In
addition, Figure 4 shows the trend in the topic share θ̄gk for each generation and the evaluation foreach of the five aspects for each topic. Table 5 shows the topic share, satisfaction value, ranking, andfrequent words. For other subscore values, the relationships between topic shares and generations areshown in Figure 4.
Based on the frequently observed topics for each generation shown in Table , we find that many
generations have at least one good topic and at least one bad topic. Further, the top topics graduallychange from iPhone 3G, launched in the Japanese market in 2008, to iPhone X, launched in 2017. This14

implies that the environmental factors have significantly changed during this tenyear period. Table
6 shows the correlation coeﬃcient matrix of the topic shares of each generation. This shows that
correlation coeﬃcients tend to be higher the closer they are to the diagonal elements of the matrix.
This implies that the same topic is likely to appear in close generations, and topics are diﬀerent for
distant generations.
For example, No.53 is a frequent topic in 3G, 3GS, and 4. In Figure 4, in terms of the topic shareθ̄g , 3G has the highest value, followed by those of 3GS and 4. Further, as the generation changes, thetopic share decreases. Additionally, we find that design and screen are highly evaluated. However, thebattery evaluation is lower than that of the other subscores. This implies that this topic was reviewed bycomparing the feature phones popular at the time. Feature phones have a very long continuous standby
time; on the other hand, the iPhone uses more electric power than its rivals. Therefore, its performanceis relatively inferior in terms of battery. However, large screens and sophisticated designs have proved
popular with consumers. Reflecting this, the overall evaluation was also higher. In addition, the reason
this topic has not appeared much after the iPhone 4s is that the diﬀusion rate of smartphones sharply
increased from 2011 to 2012. When the iPhone 4s was released, many competing smartphone models
were released, and feature phones were no longer comparison targets at the time. According to the
Ministry of Internal Aﬀairs and Communications of Japan (2015), the diﬀusion rate of smartphones atthe end of 2010 was 9.7%, but it rapidly increased to 29.3% by the end of 2011 and reached 49.5% by
the end of 2012.

15

Gen/No.
iPhone 3G
53
48

Topic Share

Satisfaction

Frequent words

0.044
0.034

0.947 (11)0.822 (91)41
iPhone
3GS
53
41
88

0.032

0.616 (105)

See Table 3
function, cellphone, usable (-1), able (-1), 1seg, flaw, drop-down, Safari,review, ease-of-use
See Table 4

0.036
0.025
0.018

0.947 (11)0.616 (105)0.898 (50)

See Table 3
See Table 4
iTunes, computer, software, synchronization, able, data, Mac, cooperation, PC, convenient

iPhone 4
53
34

0.02
0.019

0.947 (11)0.954 (7)

38

0.018

0.912 (40)

See Table 3
problem (-1), mobile, design, response, conversion, sound-quality, display,
telephone-call, operation, letter
able, setting, ringtone, simple, app, able (-1), folder, detailed, worry, dictionary

iPhone 4S
32

0.022

0.954 (5)

17

0.018

0.931 (24)

15

0.018

0.951 (8)

0.028
0.019
0.019

0.954 (6)0.951 (8)0.954 (5)

See Table 3
See above
See above

0.024

0.755 (99)

0.02
0.018

0.931 (24)0.921 (35)normal, design, mobile, response, display, letter, operation, telephone-call,sound-quality, conversion
See above
compare, feeling, have, like, slightly, personal, here, inferior, think, special

0.02
0.02
0.019

0.755 (99)0.951 (8)0.918 (37)

See above
See above
MNP, cheap, contract, fee, lump-sum, plan, basis, monthly, cancellation,
trade-in

0.03
0.02
0.019

0.85 (77)0.705 (100)0.841 (86)

See Table 3
See Table 4
size, operation, small, one-hand, screen, large, best, 4-inches, screen-size,
5-inches

0.021
0.02
0.019

0.705 (100)0.951 (8)0.85 (77)

See Table 4
See above
See Table 3

0.039
0.031

0.841 (86)0.838 (88)

100

0.024

0.901 (48)

See above
SIM-free, SIM, carrier, operate, able, usable, overseas, price, MVNO,
MVNO-SIM
design, telephone-call, mobile, response, conversion, letter, sound-quality,
music, display, menu

iPhone 77

0.026

0.88 (61)

91

0.021

0.881 (60)

40

0.021

0.866 (70)

iPhone 8
109

0.026

0.873 (64)

97

0.023

0.94 (17)

13

0.022

0.842 (82)

iPhone X
57

0.13

0.821 (92)

34
9

0.016
0.016

0.954 (7)0.705 (100)iPhone 550
1532
iPhone 5c103
1782
iPhone 5s
1031520
iPhone 671
964
iPhone 6s
91571
iPhone SE
64
35

conversion, design, letter, display, operation, mobile, response, menu,
telephone-call, buttonmusic, design, conversion, sound-quality, response, letter, display,
telephone-call, mobile, menu
pretty, simple, telephone-call, design, music, conversion, letter, operation,
response, sound-quality

waterproof, after, other-than, usable, migration, correspondence, Suica,
mobile-wallet, card, convenient
music, earphone, speaker, sound-quality, listen, playback, hear, attached,jack, as-suchchange (-1), change, evolution, improvement, increase, change, this-time,feel, rise, up
camera, photo, video, image-quality, photograph, take-a-phonto, pretty,
see, function, takespeed, processing, resolution, rise, experience, ability, strongest, current,
current, street
Plus, model, correspondence, same, VoLTE, change, disappointment, rosegold, next, personal
screen, ID, touch, button, lock, home, certification, display, unlock, disappear
See above
See Table 4

Topic share is the value of θ̄g , and satisfaction is the value of ψkJ . The numbers in parentheses after topic
share and satisfaction indicate the rankings. See Table 3 and See Table 4 indicate that the frequent words ofthe topic are listed in the suggested table. See above indicates that the topic is already listed at the top of
the table.

Table 5: Popular Topics for Each Segment
16

theta

0.025

0.015

0.005

0.016

0.010

0.004

theta

theta

0.020

0.014

0.008

0.12

0.08

0.04

0.00

theta

●

●

●

●
● ●
● ●
●
●

Topic No. 7

●

●

●

●

●

●

● ●

●

●

●

●

●

Topic No. 20

●

●

●

● ●

●

●

●

● ●

●

●

●

Topic No. 40

●

●

●

● ● ● ● ● ●

●

●

●

●

Topic No. 57

●

●

●

●

●

●

●

●

●

●

●

● ●

Topic No. 91

●

●

●

0.0

Design0.914 (71)
Mobility
0.806 (85)
Response
0.915 (55)
Screen0.938 (55)Battery
0.707 (65)
Satisfaction
0.881 (60)0.0

Design0.846 (94)
Mobility
0.805 (87)
Response
0.904 (70)
Screen0.851 (102)Battery
0.741 (40)Satisfaction
0.821 (92)0.0

Design0.801 (104)
Mobility
0.806 (86)Response
0.96 (11)
Screen0.944 (50)Battery
0.733 (48)Satisfaction
0.866 (70)0.0Design0.955 (27)
Mobility
0.903 (20)
Response
0.926 (43)
Screen0.938 (56)Battery
0.744 (39)Satisfaction
0.918 (37)0.0

Design0.91 (75)
Mobility
0.835 (69)
Response
0.917 (51)
Screen0.922 (68)Battery
0.722 (55)
Satisfaction
0.88 (61)

1.0

1.0

1.0

1.0

1.0

●

●

●

● ●
●

●

●

●

●

Topic No. 9
●
● ●

●

●

●

●

●
●
●

● ●
● ●

Topic No. 32

●

●

●

●

●
●
● ● ● ● ● ●● ● ●Topic No. 41

●

● ●

●
● ●

●

●
●

●

Topic No. 64●●●● ● ● ● ●
● ● ●

●

●

Topic No. 97

●

●

●

0.0

Design0.955 (25)
Mobility
0.918 (12)Response
0.964 (10)
Screen0.965 (17)Battery
0.754 (35)Satisfaction
0.94 (17)0.0Design0.838 (98)
Mobility
0.789 (89)Response
0.902 (73)
Screen0.878 (91)Battery
0.747 (38)Satisfaction
0.841 (86)0.0

Design0.836 (99)
Mobility
0.424 (108)
Response
0.612 (108)
Screen0.859 (99)Battery0.254 (110)Satisfaction
0.616 (105)0.0

Design0.97 (7)Mobility
0.935 (5)
Response
0.973 (7)Screen0.975 (10)Battery
0.776 (19)Satisfaction
0.954 (5)0.0

Design0.609 (108)
Mobility
0.505 (107)
Response
0.881 (80)
Screen0.892 (85)Battery
0.728 (50)
Satisfaction
0.705 (100)

1.0

1.0

1.0

1.0

1.0

● ● ● ● ● ● ●

●

●

●

Topic No. 13

●

●

●

●

●

●

●

●

●
●

● ●

● ●

Topic No. 34

●

●

●

●
●
●
●
●

● ● ●
● ● ● ●

Topic No. 48

●

●
● ● ●

●
●

●

●

●

Topic No. 71

●

● ●

●

●
●

●

●
●

●

●

●

●

Topic No. 100

●

●

●

0.0

Design0.903 (79)
Mobility
0.865 (45)
Response
0.95 (27)
Screen0.934 (58)Battery
0.736 (45)Satisfaction
0.901 (48)0.0

Design0.763 (106)
Mobility
0.552 (106)
Response
0.942 (30)
Screen0.959 (27)Battery
0.846 (3)Satisfaction
0.85 (77)0.0

Design0.974 (5)Mobility
0.781 (90)Response
0.617 (106)
Screen0.951 (37)Battery
0.393 (106)Satisfaction
0.822 (91)0.0

Design0.958 (20)
Mobility
0.898 (25)
Response
0.977 (4)
Screen0.979 (6)Battery
0.824 (7)Satisfaction
0.954 (7)0.0

Design0.832 (101)
Mobility
0.708 (101)Response
0.908 (66)
Screen0.892 (84)Battery
0.78 (17)Satisfaction
0.842 (82)

1.0

1.0

1.0

1.0

1.0

theta

theta

theta

●
●

●

●

●

●

●

●

●

●

Topic No. 15

●
●
●

●

● ●

●
●

● ● ●

●

●

Topic No. 35

●

●

●

●
● ●

●

●

● ●

●
● ● ● ●

Topic No. 50

●

●

●

●

●
●

●

●

●

●

●

Topic No. 82

●

●
●

●

●
● ● ●

●

●
●
●

●

Topic No. 103

●

●

●

0.0

Design0.815 (103)
Mobility
0.763 (94)
Response
0.809 (97)
Screen0.822 (104)Battery
0.617 (90)Satisfaction
0.755 (99)0.0

Design0.937 (53)
Mobility
0.87 (43)
Response
0.953 (25)
Screen0.946 (45)Battery
0.758 (32)
Satisfaction
0.921 (35)0.0

Design0.974 (4)
Mobility
0.975 (1)
Response
0.985 (1)
Screen0.982 (3)Battery
0.796 (11)Satisfaction
0.954 (6)0.0

Design0.902 (81)
Mobility
0.849 (60)
Response
0.84 (90)
Screen0.858 (100)Battery
0.726 (53)Satisfaction
0.838 (88)0.0

Design0.954 (29)
Mobility
0.915 (14)Response
0.975 (5)Screen0.974 (11)Battery
0.817 (8)Satisfaction
0.951 (8)

1.0

1.0

1.0

1.0

1.0

●

●

●

●

●

●

●

●

●

● ●

Topic No. 17
●

●

●

●

●

●
●
●

●

●

●
● ● ●

Topic No. 38

●

●

●

●

●

● ● ● ● ● ●● ● ●Topic No. 53

●

●

●
● ●
● ●

●
● ● ●

Topic No. 88

●

●

●

●

●
●
● ●

●

●

●

●

Topic No. 109

●

●

●

0.0

Design0.863 (93)
Mobility
0.73 (99)
Response
0.922 (46)
Screen0.942 (51)Battery
0.774 (22)Satisfaction
0.873 (64)0.0

Design0.959 (18)
Mobility
0.841 (67)
Response
0.861 (87)
Screen0.951 (38)Battery0.59 (93)Satisfaction
0.898 (50)0.0

Design0.985 (1)
Mobility
0.729 (100)Response
0.813 (95)
Screen0.988 (1)Battery0.429 (105)
Satisfaction
0.947 (11)0.0

Design0.961 (14)Mobility
0.846 (63)
Response
0.916 (53)Screen0.959 (29)Battery0.635 (86)Satisfaction
0.912 (40)0.0

Design0.94 (49)
Mobility
0.882 (35)
Response
0.956 (20)
Screen0.96 (24)Battery
0.767 (26)Satisfaction
0.931 (24)

1.0

1.0

1.0

1.0

1.0

Figure 4: Dynamics and Profiles of Popular Topics for Each Segment

This Figure displays the topics shown in Table 5. The line chart is the topic share (θ̄g ) in each generation. The bar graph on the right of each topic shows the value of ψ. The lighter
the colors are, the higher the rank is (it has a relatively large value), and the darker the colors are, the lower the rank is (it has a relatively small value).

theta

0.020

0.014

0.008

3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X

3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X

3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X

3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X

3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X

0.020
0.015
0.010
0.010 0.015 0.020
0.025
0.015
0.005

theta
theta
theta
theta

0.035
0.020
0.005
0.015
0.005

3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X
3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X
3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X
3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X
3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X

0.015
0.005
0.016
0.012
0.008
0.035
0.025
0.015
0.005

theta
theta
theta
theta

0.030
0.020
0.010
0.010 0.015 0.020

3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X
3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X
3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X
3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X
3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X

0.020
0.014
0.008
0.025
0.015
0.005
0.025
0.015
0.005

theta
theta
theta
theta

0.016
0.012
0.008
0.020
0.010

3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X
3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X
3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X
3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X
3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X

0.016
0.010
0.018
0.014
0.010
0.006
0.03

theta
theta
theta
theta
theta

0.01
0.020
0.010
0.020
0.010

3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X
3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X
3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X
3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X
3G
3GS
4
4S
5
5c
5s
6
6s
SE
7
8
X

17

iPhone 3G
iPhone 3GS
iPhone 4
iPhone 4S
iPhone 5
iPhone 5c
iPhone 5s
iPhone 6
iPhone 6s
iPhone SE
iPhone 7
iPhone 8
iPhone X

3G
1.00
0.87
0.54
0.21
-0.02
-0.02
-0.04
-0.07
-0.11
-0.08
-0.15
-0.13
-0.08

3GS

4

4S

5

5c

5s

6

6s

SE

7

8

X

1.00
0.74
0.41
0.11
0.19
0.09
0.05
0.00
0.03
-0.05
-0.06
-0.07

1.00
0.76
0.42
0.50
0.38
0.28
0.28
0.15
0.16
0.09
-0.03

1.00
0.66
0.72
0.66
0.41
0.45
0.32
0.26
0.24
-0.05

1.00
0.68
0.73
0.43
0.36
0.24
0.23
0.18
-0.04

1.00
0.77
0.55
0.52
0.48
0.37
0.32
0.01

1.00
0.58
0.61
0.53
0.37
0.36
0.17

1.00
0.81
0.55
0.51
0.51
0.13

1.00
0.53
0.63
0.64
0.37

1.00
0.32
0.45
0.07

1.00
0.70
0.26

1.00
0.31

1.00

Table 6: Correlation Matrix of Topic Share

5.4

Share of Salient Topics

In this section, we examine the factors that increase or decrease the topics from the basic information
obtained from each review on salient topics mentioned in the previous section. The parameter θd
is obtained for each∑document. We set K = 110; therefore, θd is a 110-dimensional vector and hasK
the property thatk=1 θdk = 1. Following Puranam, Narayan, and Kadiyali (2017), we conduct
a regression analysis to assess the precedence of θd . There are some models that assume objective
variables such as this. For example, Cooper and Nakanishi (1988) and Berry (1994) proposed models
that analyze market share. We apply a model developed by Berry (1994), which assumes a logit modelfor the consumers’ choice probabilities, which are components of market share. Berry (1994) assumes
an external good and subtracts it from the market share of each alternative to obtain a linear model.
Therefore, in this study, the share of the 85 topics excluding the 25 salient topics shown in Table 5 andFigure 4 in the previous section is regarded as the share of external goods. We examine the relative
impact of each salient topic compared with other topics with a high share. Let the total share of 85
topics in review d be denoted by θd0 and the basic information attached to each review denoted by Xd .
We estimate the following linear regression model:ln(θdk ) − ln(θd0 ) = Xd λk + εdk , εdk ∼ N (0, σk2 )(27)

In Xd , the following explanatory variables and control variables are used:• Latest Model: As many new smartphone models are released during the analysis period, existingmodels become relatively obsolete. Therefore, a dummy variable that takes the value 1 if thereviewed model was the latest model when the review was posted, and 0 otherwise, is included in
the explanatory variables. If the estimated coeﬃcient is positive, the topic is likely to be a reviewof the latest model, and if the parameter is negative, it is likely to be a review of an outdated
model. Out of 5,375 reviews, 4,193 (78 %) reviews were for the latest models at the time of
posting, whereas 1,182 (22 %) reviews were for outdated models.
• Reviewer’s Experience: Each review includes the author’s name; therefore, we can track if theauthor has posted reviews in the past. Based on the name, for each review, a dummy variable
that takes the value 1 if the author has posted a review of the iPhone in the past, and 0 otherwise,is included in the explanatory variables. If the estimated coeﬃcient is positive, the topic is likely
to be mentioned by consumers who have posted reviews in the past, and if the parameter isnegative, it is likely to be mentioned by consumers who have posted reviews more recently.
• Year Control: The regression model includes the year a review was posted as a control variable.
Since time trends may not change linearly, a separate dummy variable is included for each year.
In the estimation, we introduce the constraint that the parameter for 2008 (the first year) is 0.

18

• Generation Control: We also include a dummy variable corresponding to the smartphone model
to be reviewed as a control variable. In the estimation, we introduce the constraint that theparameter for 3G (the first model) is 0.

0.6

Figure 5 shows the obtained results, specifically, the estimated coeﬃcients of latest model andreviewer’s experience. The horizontal axis shows the estimated coeﬃcients of latest model, and thevertical axis those of reviewer’s experience. The numbers in the figure are topic numbers, and thesymbols attached as subscripts of the numbers indicate the significance level of latest model/reviewer’s
experience. If the symbol is “+” or “−”, the estimated coeﬃcient is significant at the 5% level; “o”
indicates that it is not significant.

0.2

13 o/+
35 −/+17 o/+
64 −/o0.0

Reviewers' Experience

0.4

40 o/+

50 o/+

o/+
20 97
o/+
82
o/+
100o/+
o/oo/o
10957
34
o/o
71o/o
9 o/o91 o/o
32 o/o

o/o
38
7 o/o

41 o/o88 o/−
48 o/−

15 o/−−0.2

103o/−
53 +/−

−0.2

−0.1

0.0

0.1

0.2

0.3

Latest Model

Figure 5: Result of Regression Analyisis

The results in Figure 5 can be better interpreted by comparing them with those in Table 5 andFigure 4. For example, if more reviews of outdated models were posted, the appearance probability of
topic No.35 would increase. Furthermore, consumers who have previously posted reviews of the iPhone
would also increase the likelihood of this topic. Looking at Table 5, No.35 is a topic about SIM-free and
MVNO (Mobile Virtual Network Operator, a new low-priced carrier), and it seems to include reviews by
consumers relatively familiar with telecommunication policies and technologies. This implies that thereviewers have obtained the iPhone at a low price by purchasing a model when it has become outdated
and the price has dropped. Looking at Figure 4, the probability that this topic appears would increase
for reviews of the iPhone SE, which was released as a low-priced version. This suggests that the reviews
are by consumers who want to acquire smartphones at a low price. In addition, the results show that
No.40 is more likely to consist of reviews by consumers who have posted reviews in the past. Looking
at Figure 4, the relationship of this topic with the iPhone 7, 8, and 6s is particularly strong. Therefore,
this topic seems to have been created by consumers who had been using an older model before the timeof their review and then bought a new model and reviewed it.

6Conclusion

In this study, we developed the TSAM, expanding the commonly used LDA method and applied it
to online reviews that have both document information written in natural language and evaluation19

information in the form of continuous scores, to classify natural language and score information simultaneously. In the empirical analysis, the TSAM was applied to 10 years of review data on the iPhone
3G through the iPhone X in the Japanese market. The following results were obtained:
First, we found that the TSAM can be used as a prediction model. We cannot use LDA for prediction
because LDA is an unsupervised model. However, using the TSAM proposed in this study, continuousscores are predicted from document data written in natural language. In particular, as a result of acomparison with regression models (an ordered logit model and a binomial logit model), the TSAM’sprediction performance when only document information was obtained was very high. The TSAM canbe used as a prediction model using a large amount of document information as input data. Further,in terms of computational eﬃciency, since collapsed Gibbs sampling is used for parameter estimation,
fast and stable results can be obtained. Second, by interpreting the model, it is possible to obtain more
useful information for marketing decision-making compared to that obtained from LDA. Although LDA
cannot directly interpret the sentiment and mood of a document, the TSAM provides an average ratingscore for each topic. Therefore, we can directly determine whether the topic itself corresponds to agood rating or a bad one. Furthermore, our model provides parameter estimates and expectations
for five aspects of a product. Although the data adapted in this study include multiple evaluation
aspects and there may be large diﬀerences in the scores across the diﬀerent aspects, we can visuallyinterpret the relationship between topics and aspects based on the output of the model. In addition, by
checking words that are strongly related to the topic, researchers can find factors aﬀecting each scorethat cannot be discovered by simply looking at the evaluation score of each aspect. Furthermore, thefactors aﬀecting topic shares were also examined using a market share model.
Here, we would like to identify some issues with this study and propose a direction for future
research. The first issue is that the examination of the factors aﬀecting each topic is limited in thisstudy. Some previous studies have proposed a model that can examine precedence by assuming a linear
combination regression term for the parameters of the Dirichlet distribution. However, it is diﬃcult toassume conjugate prior distributions for the prior parameters of this linear combination. This implies
that the estimation result will be ineﬃcient and unstable. In this study, since the model is developed
assuming only conjugate priors, we cannot incorporate linear combination priors. The second issue isthe number of topics. In marketing decision-making, it is better to discuss a number of topics thatcan be interpreted by analysts. In the K = 110 model, it is not easy to interpret all the topics. As afuture research topic, we could try to use a hierarchical model such as Pachinko allocation model (Liand MaCallum, 2006). As regards the practical range, it is necessary to analyze not only the online
reviews of the smartphones covered in this study but also those of other smartphones and to verify theproperties of the TSAM.

A

SAM Simulation Results

In this section, we examine the SAM, comparing it with another well-known method. As defined above,
SAM requires J-a dimensional integer score vector as input data. Each score needs to be evaluated on
the Qj point scale, but the range does not have to be the same for all items, and the binomial value{0, 1} scale is also acceptable. In this section, we aim to assess the results by comparing them withthose from another method. Therefore, we simulate input data and compare the results. The method
we compare the SAM with is k-means, which is frequently used for clustering. Two-dimensional datawith D = 400 are generated as sample data. All the data are integers from 0 to 99. We assume there
are four groups, with the following average values: (70, 70), (30, 70), (70, 30), (30, 30). There are 100
observations in each group. Therefore, the settings for SAM are J = 2, (Q1 , Q2 ) = (100, 100).
To compare the two methods, we focus on the SAM parameter {ψkj }. This parameter corresponds
to the cluster mean in the k-means method. For SAM, the first 1,000 iterations are omitted as burn-in(h)(h)
and the subsequent 1,000 iterations are collected as the sample. ψkj is obtained from cd in each(h)iteration, and the median of ψkj is used for the analysis. In addition, we classify each observation into(h)

one of four topics by using the topic allocation parameter cd . Figure 6 shows the results of the k-means

20

clustering (left) and the expected values Qj ψkj (right). It can be seen that almost the same values are
obtained from both methods. This implies that the proposed TSAM combines LDA and k-means.

100

Result of SAM

100

Result of k−means

●

80

80

●
●
●
●

●

60
●
●

●

●
●

●
● ● ●
● ● ●●● ●● ●●●● ● ●
●
●
●
●● ●
● ●
●
●
● ●●
●● ● ●●●●●● ●● ●●●● ● ●
●●● ●
●
●
●
●
●
●
●●●
●
●
● ●
●
●
●● ● ● ●
●
●
● ●●

x2

●

●

40

●

●

20

20

●

●●
●

●●

●

60
x2
40

●

● ●
● ● ●●●●●●● ●● ●●● ● ●●●● ● ●● ●●● ●
●
●●●●
●
●
●
●
● ●
● ●● ● ●●● ●● ●●● ●●●●● ● ● ●●●●● ● ●●
●● ●
●●
●
●
●●
●●
●
● ●
●
●●
●
●
●
●

●
●

●

●

●●
●

0

0

●

0

20

40

60

80

100

0

20

40

x1

Figure 6: Result of k-means and SAM−3200000−3600000

log Likelihood

Sampling Path of TSAM

−4000000

B

60
x1

0

500

1000

1500

2000

2500

3000

Figure 7: Sampling Path of TSAM K = 110

21

80

100

References
Anderson, E. T., & Simester, D. I. (2014). Reviews without a purchase: Low ratings, loyal customers,
and deception. Journal of Marketing Research, 51(3), 249-269.
Ansari, A., Essegaier, S., & Kohli, R. (2000). Internet recommendation systems. Journal of Marketing
Research, 37(3), 363-375.
Balducci, B., & Marinova, D. (2018). Unstructured data in marketing. Journal of the Academy ofMarketing Science, 46(4), 557-590.
Berger, J., Humphreys, A., Ludwig, S., Moe, W. W., Netzer, O., & Schweidel, D. A. (2019). Uniting
the Tribes: Using Text for Marketing Insight. Journal of Marketing.
Berger, J., & Milkman, K. L. (2012). What makes online content viral?. Journal of marketing research,
49(2), 192-205.
Berry, S. T. (1994). Estimating discrete-choice models of product diﬀerentiation. The RAND Journalof Economics, 25(2), 242-262.
Blei, D. M., & Laﬀerty, J. D. (2006). Dynamic topic models. In Proceedings of the 23rd international
conference on Machine learning (pp. 113-120). ACM.
Blei, D. M., & Laﬀerty, J. D. (2007). A correlated topic model of science. The Annals of Applied
Statistics, 1(1), 17-35.
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of machine Learning
research, 3(Jan), 993-1022.
Borah, A., & Tellis, G. J. (2016). Halo (spillover) eﬀects in social media: do product recalls of one
brand hurt or help rival brands?. Journal of Marketing Research, 53(2), 143-160.
Büschken, J., & Allenby, G. M. (2016). Sentence-based text analysis for customer reviews. Marketing
Science, 35(6), 953-975.
Chen, Z., & Lurie, N. H. (2013). Temporal contiguity and negativity bias in the impact of online wordof mouth. Journal of Marketing Research, 50(4), 463-476.
Chevalier, J. A., & Mayzlin, D. (2006). The eﬀect of word of mouth on sales: Online book reviews.
Journal of marketing research, 43(3), 345-354.
Cooper, L. G., & Nakanishi, M. (1988). Market Share Analysis, International Series in Quantitative
Marketing, Boston: Kluwer Academic Publishers.
Dellarocas, C., Zhang, X. M., & Awad, N. F. (2007). Exploring the value of online product reviews in
forecasting sales: The case of motion pictures. Journal of Interactive marketing, 21(4), 23-45.
Floyd, K., Freling, R., Alhoqail, S., Cho, H. Y., & Freling, T. (2014). How online product reviews aﬀect
retail sales: A meta-analysis. Journal of Retailing, 90(2), 217-232.
Griﬃths, T. L., & Steyvers, M. (2004). Finding scientific topics. Proceedings of the National academyof Sciences, 101(suppl 1), 5228-5235.
Hewett, K., Rand, W., Rust, R. T., & Van Heerde, H. J. (2016). Brand buzz in the echoverse. Journalof Marketing, 80(3), 1-24.
Humphreys, A., & Wang, R. J. H. (2017). Automated text analysis for consumer research. Journal ofConsumer Research, 44(6), 1274-1306.

22

Kannan, P. K., & Li. H. A. (2017). Digital marketing: A framework, review and research agenda.
International Journal of Research in Marketing, 34(1), 22-45.
Kudo, T., Yamamoto, K., & Matsumoto, Y. (2004). Applying conditional random fields to Japanesemorphological analysis. In Proceedings of the 2004 conference on empirical methods in natural language processing (pp. 230-237).
Lee, T. Y., & Bradlow, E. T. (2011). Automated marketing research using online customer reviews.
Journal of Marketing Research, 48(5), 881-894.
Li, W., & McCallum, A. (2006). Pachinko allocation: DAG-structured mixture models of topic correlations. In Proceedings of the 23rd international conference on Machine learning (pp. 577-584).
ACM.
Liu, X., Burns, A. C., & Hou, Y. (2017). An investigation of brand-related user-generated content on
Twitter. Journal of Advertising, 46(2), 236-247.
Liu, Y. (2006). Word of mouth for movies: Its dynamics and impact on box oﬃce revenue. Journal ofmarketing, 70(3), 74-89.
Luo, X. (2009). Quantifying the long-term impact of negative word of mouth on cash flows and stock
prices. Marketing Science, 28(1), 148-165.
Mimno, D., Wallach, H. M., Naradowsky, J., Smith, D. A., & McCallum, A. (2009). Polylingual topic
models. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing:Volume 2-Volume 2 (pp. 880-889). Association for Computational Linguistics.
Nam, H., Joshi, Y. V., & Kannan, P. K. (2017). Harvesting brand information from social tags. Journalof Marketing, 81(4), 88-108.
Onishi, H., & Manchanda, P. (2012). Marketing activity, blogging and sales. International Journal of
Research in Marketing, 29(3), 221-234.
Puranam, D., Narayan, V., & Kadiyali, V. (2017). The eﬀect of calorie posting regulation on consumer
opinion: A flexible latent Dirichlet allocation model with informative priors. Marketing Science, 36(5),726-746.
Rosen-Zvi, M., Griﬃths, T., Steyvers, M., & Smyth, P. (2004). The author-topic model for authorsand documents. In Proceedings of the 20th conference on Uncertainty in artificial intelligence (pp.
487-494). AUAI Press.
Sato, T., Hashimoto, T., & Okumura, M. (2017). Implementation of a word segmentation dictionary
called mecab-ipadic-NEologd and study on how to use it eﬀectively for information retrieval. In
Proceedings of the twenty-three annual meeting of the association for natural language processing.
The Association for Natural Language Processing.
Stephen, A. T., & Galak, J. (2012). The eﬀects of traditional and social earned media on sales: A studyof a microlending marketplace. Journal of marketing research, 49(5), 624-639.
Tirunillai, S., & Tellis, G. J. (2012). Does chatter really matter? Dynamics of user-generated content
and stock performance. Marketing Science, 31(2), 198-215.
Tirunillai, S., & Tellis, G. J. (2014). Mining marketing meaning from online chatter: Strategic brandanalysis of big data using latent dirichlet allocation. Journal of Marketing Research, 51(4), 463-479.
Toubia, O., Iyengar, G., Bunnell, R., & Lemaire, A. (2019). Extracting Features of Entertainment
Products: A Guided Latent Dirichlet Allocation Approach Informed by the Psychology of Media
Consumption. Journal of Marketing Research, 56(1), 18-36.

23

Trusov, M., Ma, L., & Jamal, Z. (2016). Crumbs of the cookie: User profiling in customer-base analysis
and behavioral targeting. Marketing Science, 35(3), 405-426.
Wedel, M., & Kannan, P. K. (2016). Marketing analytics for data-rich environments. Journal of Marketing, 80(6), 97-121.

24

A

Online Appendix: Estimation of TSAM

A.1

Model Definition

Assume that there are D documents and N words (D < N ). There are V vocabulary words observedin the text data; therefore, word wi is a V -dimensional vector whose element wiv = 1 if the i-th word
is vocabulary word v; otherwise, wiv′ = 0, ∀v ′ ̸= v. wi is aﬃliated with any one of the documents;
therefore, we introduce a document aﬃliation index xi . If wi if aﬃliated with document d, xid = 1,
and xid′ = 0, ∀d′ ̸= d. In addition, for each document, J item ratings are observed. L let ydj be theordered integer score for aspect j in document d, where the minimum score is 0 and the maximum scoreis Qj − 1. In addition, assume there are K latent topics and denote the topic aﬃliation index for text
as zi and the score aﬃliation index as cd .
K∏

wi ∼ CategoricalV (ϕ̃i ), ϕ̃iv =ϕzkvik

(28)

xid
θdk

(29)

k=1

zi ∼ CategoricalK (θ̃i ), θ̃ik =

D
∏
d=1

K
∏

ydj ∼ Binomial(Qj , ψ̃dj ), ψ̃dj =

c

ψkjdj

(30)

k=1

cd ∼ CategoricalK (θd )(31)

The prior distributions of ϕk , ψd and θd are defined as follows:ϕk∼DirichletV (β)

(32)

θd

∼

DirichletK (α)

(33)

ψkj

∼

Beta(γ)

(34)
(35)

A.2

Densities
π(W |Φ, Z)=N∏

π(wi |Φ)

i=1

=

=

N ∏
V
∏

(K
∏

i=1 v=1

k=1

K
V ∏
N ∏
∏

(36)
)wiv
ϕzkvik(37)

ϕzkvik ×wiv

(38)

i=1 v=1 k=1π(Z|Θ) =N∏

π(zi |Θ)

i=1

=

=

N ∏
K
∏

(D
∏

i=1 k=1

d=1

N ∏
K ∏
D
∏
i=1 k=1 d=1

25

(39)
)zik
xid
θdk

(40)

xid ×zikθdk(41)

To define the density of Y , we introduce the sub-observation ydjq . For example, if Qj = 4 andydj = 2, we divide ydj into the Qj -dimensional observation vector {ydj1 , ydj2 , ydj3 , ydj4 } = {1, 1, 0, 0}.
ydjq is the q-th element of the vector.

π(Y |Ψ, C) =

Qj
D ∏
J ∏
∏
d=1 j=1 q=1==π(ydjq |Ψ, C)

Qj
D ∏
J ∏
∏

(K
∏

d=1 j=1 q=1

k=1

Qj K (
D ∏
J ∏
∏
∏

cdk
ψkj

(42)

)ydjq ( K∏

)1−xdjq[1 − ψkj ]cdk

(43)

k=1
y

1−ydjq(ψkj ) djq (1 − ψkj )

)cdk
(44)

d=1 j=1 q=1 k=1(45)π(C|Θ) =D∏

π(cd |Θ)

(46)

d=1

=

D ∏
K
∏

c

(θdk ) dk

(47)

cdk
θdk

(48)

d=1 k=1=KD ∏∏d=1 k=1

π(Φ)

=

K
∏

π(ϕk )

(49)

k=1

=

K
∏
k=1

∝

1 ∏ β−1ϕ
B(β) v=1 kv

K ∏
V
∏

V

ϕβ−1
kv

(50)

(51)

k=1 v=1π(Ψ) =K ∏J∏

π(ψkj )

(52)

k=1 j=1

=

J
K ∏
∏
k=1

∝

1
γ−1
ψkj
(1 − ψkj )γ−1
B(γ)
j=1

K ∏
J
∏

γ−1
ψkj
(1 − ψkj )γ−1

k=1 j=1

26

(53)

(54)

π(Θ)

D
∏

=

π(θd )

(55)

d=1
D
∏

=

d=1

1 ∏ α−1
θdk
B(α)
K

D ∏
K
∏

∝

(56)

k=1

α−1
θdk

(57)

d=1 k=1

A.3

Likelihood Function
L(D|Z, C, Φ, Ψ) =π(W, Y |Φ, Ψ, Z, C)(58)= π(W |Φ, Z)π(Y |Ψ, C)
=

N ∏
V ∏
K
∏

ϕzkvik ×wiv ×

i=1 v=1 k=1

A.4

(59)
D ∏
J ∏ ∏
K (
∏
Qj

y

1−ydjq(ψkj ) djq (1 − ψkj )

)cdk
(60)

d=1 j=1 q=1 k=1

Full Conditional Posterior Distribution
π(Z, C, Φ, Ψ, Θ|D) ∝ L(D|Z, C, Φ, Ψ)π(Z|Θ)π(C|Θ)π(Φ)π(Ψ)π(Θ)
Qj K (
N ∏
V ∏
K
D ∏
J ∏
)cdk
∏
∏
∏
y
1−y
zik ×wiv∝
ϕkv
×(ψkj ) djq (1 − ψkj ) djq
i=1 v=1 k=1

×

N ∏
K ∏
D
∏

K ∏
V
∏

xid ×zik
θdk
×

∝

D ∏
K
∏

cdk
θdk

(63)

d=1 k=1
K ∏
J
∏

ϕβ−1
kv ×

k=1 v=1
K
V ∏
∏

(62)

d=1 j=1 q=1 k=1

i=1 k=1 d=1

×

(61)

k=1 j=1

∑N

z

D ∏
K
∏

γ−1
ψkj
(1 − ψkj )γ−1 ×

α−1
θdk

(64)

d=1 k=1w

ϕkvi=1 ik iv(65)

v=1 k=1

×

J ∏
K
∏

∑D

(ψkj )∑ Qj

d=1

q=1 ydjq cdk

x z

D ∏
K
∏

∑D

(1 − ψkj )

∑Qj

d=1

q=1 (1−ydjq )cdk

(66)

j=1 k=1

×

D
K ∏
∏

∑N

θdki=1 id ik ×K ∏V∏
k=1 v=1(67)

d=1 k=1

k=1 d=1

×

cdk
θdk

ϕβ−1
kv ×

K ∏
J
∏
k=1 j=1

27

γ−1
ψkj
(1 − ψkj )γ−1 ×D ∏K∏
d=1 k=1

α−1
θdk

(68)

To simplify the expression, we define the following variables:
nkv

=

N
∑

zik wiv

(69)

i=1

nkj1

=

Qj
D ∑
∑

ydjq cdk(70)(1 − ydjq )cdk

(71)

d=1 q=1

nkj0

=

Qj
D ∑
∑
d=1 q=1

ndk

=

N
∑

xid zik(72)

i=1π(Z, C, Φ, Ψ, Θ|D) ∝

V ∏
K
∏

ϕnkvkv ×

v=1 k=1

×

J ∏
K
∏

K ∏
D
∏

K ∏
V
∏

D ∏
K
∏

ndk
θdk
×

V ∏
K
∏

(73)

cdk
θdk

(74)

d=1 k=1

ϕβ−1kv ×

k=1 v=1∝n

j=1 k=1

k=1 d=1

×n(ψkj ) kj1 (1 − ψkj ) kj0

K ∏
J
∏

α−1
ψkj
(1 − ψkj )α−1 ×

k=1 j=1

D ∏
K
∏

γ−1
θdk

(75)

d=1 k=1

ϕnkvkv +β−1(76)

v=1 k=1

×

J ∏
K
∏

n

(ψkj ) kj1

+γ−1

n

(1 − ψkj ) kj0

+γ−1

(77)

j=1 k=1

×

K ∏
D
∏

ndk +cdk +α−1θdk

k=1 d=1

28(78)

The TSAM parameters Z and C are obtained by collapsed Gibbs sampling.
∫ ∫ ∫π(Z, C|D) ∝π(Z, C, Φ, Ψ, Θ|D)dΦdΨdΘ
∝

∫ ∏
V ∏
K

(79)

ϕnkvkv +β−1 dΦ(80)

v=1 k=1

×

∫ ∏
J ∏
K

n

(ψkj ) kj1

+γ−1

n

(1 − ψkj ) kj0

+γ−1

dΨ

(81)

j=1 k=1

×

∫ ∏
K ∏
D

ndk +cdk +α−1
θdk
dΘ

(82)

k=1 d=1∏VK∏
v=1 Γ(nkv + β)∝∑V
v=1 nkv + V β)
k=1 Γ(×J ∏K∏
Γ(nkj1 + α)Γ(nkj0 + γ)
j=1 k=1

×Γ(nkj1 + nkj0 + 2γ)

(83)

(84)

∏K

D
∏

k=1 Γ(ndk + cdk + α)∑K
k=1 (ndk + cdk ) + Kα)
d=1 Γ((85)

Let us introduce the following variables:
nk

=

V
∑

nkv

(86)

nkj1 + nkj0

(87)

v=1

nkj
nd

=
=

K
∑

ndk

(88)

k=1

Note that∑K

k=1 cdk = 1.

A.5

Posterior Distributions

A.5.1

Conditional Posterior Distribution of zik

The conditional posterior distribution of zik is similar to that of the original LDA.
()()
nkv∗ ,−i + βnd∗ k + cdk + γπ(zik = 1|Z−i C, D) ∝nk,−i + V β
nd∗ ,−i + 1 + Kγ()
nkv∗ ,−i + β∝(nd∗ k,−i + cdk + γ)
nk,−i + V β

29

(89)
(90)

A.5.2

Conditional Posterior Distribution of cdk
J ∏K∏Γ(nkj1 + α)Γ(nkj0 + α)π(cdk = 1|Z, C−d , D) ∝j=1 k=1

×

D
∏

(91)

Γ(nkj1 + nkj0 + 2α)∏K

k=1 Γ(ndk + cdk + γ)∑K
k=1 (ndk + cdk ) + Kγ)
d=1 Γ((92)

J∏
Γ(nkj1 + α)Γ(nkj0 + α)

∝

j=1

(93)

Γ(nkj1 + nkj0 + 2α)Γ(ndk + cdk + γ)
(94)∑K
Γ( k=1 (ndk + cdk ) + Kγ)∑D∑D ∑Qj∑D∑D ∑Qjcdk (1−ydjq ) = d=1 cdk (Qj −ydjq cdk = d=1 cdk ydj and nkj0 = d=1 q=1
where nkj1 = d=1 q=1∑Dydj ). Therefore, nkj1 + nkj0 = d=1 cdk Qj and we can obtain the following equations:D∑Γ(nkj1 + α) = Γ 
cd′ k yd′ j + cdk ydj + α
(95)
×



d′ =1,d′ ̸=d
D
∑

= Γ


cd′ k yd′ j + ydj + α(96)d′ =1,d′ ̸=d= Γ (nkj1,−d + ydj + α)
ydj
∏

=

(97)

(nkj1,−i + q − 1 + α) Γ (nkj1,−d + α)

(98)

q=1

(99)
where nkj1,−d =∑Dd′ =1,d′ ̸=d cd k yd j .
′Γ(nkj0 + α)

′

=





D
∑

Γ

cd′ k (Qj − yd′ j ) + cdk (Qj − ydj ) + αd′ =1,d′ ̸=d


=

D
∑

Γ

(100)


cd′ k yd′ j + (Qj − ydj ) + α(101)d′ =1,d′ ̸=d=Γ (nkj0,−d + (Q − ydj ) + α)(102)

(Qj −ydj )=∏(nkj0,−d + q − 1 + α) Γ (nkj0,−d + α)

(103)

q=1

(104)
where nkj0,−d =∑Dd′ =1,d′ ̸=d zd k (Qj − xd j ).
′Γ(nkj1 + nkj0 + 2α)

′

= Γ(

D
∑

cdk Qj + 2α)

(105)

d=1

= Γ (nkj,−d + Qj + 2α)
∏

(106)

Qj

=

(nkj,−d + q − 1 + 2α) Γ (ndk,−d + 2α)

q=1

30

(107)

∑D
where nk,−d = d′ =1,d′ ̸=d zd′ k Q.
From the equations above, we can obtain the following relationship:J∏Γ(nkj1 + α)Γ(nkj0 + α)Γ(nkj1 + nkj0 + 2α)∏ydj∏(Qj −ydj )(nkj0,−d + q − 1 + α)
q=1 (nkj1,−d + q − 1 + α)
q=1
∝
∏Qj
q=1 (nkj,−d + q − 1 + 2α)

(108)

j=1

(109)

In addition, we can transform the component as follows:Γ(ndk + cdk + γ)∑KΓ( k=1 (ndk + cdk ) + Kγ)

Γ(ndk + cdk + γ)
Γ(nd + 1 + Kγ)

∝

(110)

∝ Γ(ndk + 1 + γ)(111)

∝ (ndk + γ)Γ(ndk + γ)

(112)

The posterior distribution of cdk is obtained as follows:π(cdk = 1|Z, C−d , D)∏(Qj −ydj )∏ydj(nkj0,−d + q − 1 + α)
q=1
q=1 (nkj1,−d + q − 1 + α)∝(ndk + γ)∏Qj
q=1 (nkj,−d + q − 1 + 2α)

A.6

(113)
(114)

Parameters Ψ, Φ, and ΘGiven Z and C, the conditional posterior distribution of ψkj is a beta distribution; therefore, theexpectation is obtained as follows:π(ψkj |Z, C, Φ, Ψ−kj , Θ, D) ∝

K
V ∏
∏

nkv +β−1
ϕkv

(115)

v=1 k=1

×

J ∏
K
∏

n

(ψkj ) kj1

+γ−1

n

(1 − ψkj ) kj0

+γ−1

(116)

j=1 k=1

×∝E[ψkj ] =K ∏D∏

ndk +cdk +α−1
θdk

(117)

k=1 d=1
n
+γ−1
n
+γ−1
(ψkj ) kj1(1 − ψkj ) kj0(118)

nkj1 + γ
nkj1 + nkj0 + 2γ

(119)

Similarly, given Z and C, the conditional posterior distributions of ϕkv and θdk are Dirichlet distributions. Therefore, their respective expectations are obtained as follows:
nkv +β−1π(ϕkv |Z, C, Φ−kv , Ψ, Θ, D) ∝ ϕkv

E[ϕkv ] =nkv + βnk + V βπ(θdk |Z, C, Φ, Ψ, Θ−dk , D) ∝E[θdk ] =ndk + αnd + Kα

31

(120)
(121)

ndk +cdk +α−1
θdk

(122)
(123)

B

Online Appendix: Pseudocode

B.1

Pseudocode for LDA

Algorithm 1 Algorithm for LDA
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

Input: K (Number of topics): Integer
Input: BN (Number of burn-in iteratoins): Integer
Input: N N (Number of sampling iteratoins): Integer
Input: X (Document Index): A N -dimensional vector, X[i] ∈ {1, · · · , D}, ∀i = 1, · · · , N
Input: W (Word Index): A N -dimensional vector, W [i] ∈ {1, · · · , V }, ∀i = 1, · · · , N
Input: Z (Topic Allocation Initial Value): A N -dimensional vector, Z[i] ∈ {1, · · · , K}, ∀i = 1, · · · , N
Input: α, β: Hyperparameters
Output: ZZ: A N × K matrix
Initialisation:
define: ZZ is a N N × N matrix.
define: N kv is a K × V matrix, N k is a K-dimensional vector, N dk is a D × K matrix.
For i ← 1 to N do:v ← W [i], k ← Z[i], d ← X[i]N kv[k, v] ← N kv[k, v] + 1, N k[k] ← N k[k] + 1, N dk[d, k] ← N dk[d, k] + 1
End forCollapsed Gibbs Sampling Loop:define: p is a K-dimensional vector.
For nn ← 1 to BN + N N do:
For i ← 1 to N do:v ← W [i], k ← Z[i], d ← X[i]N kv[k, v] ← N kv[k, v] − 1, N k[k] ← N k[k] − 1, N dk[d, k] ← N dk[d, k] − 1
For k ← 1 to K do:
kv[k,v]+β(N dk[d, k] + α)p[k] ← N
N k[k]+V ∗β
End fork ∼ Cat (p)N kv[k, v] ← N kv[k, v] + 1, N k[k] ← N k[k] + 1, N dk[d, k] ← N dk[d, k] + 1
If nn > BN :ZZ[i, k] ← ZZ[i, k] + 1
End forEnd forEnd for

B.2

Online Appendix: Pseudocode for SAM (Score Allocation Model)

Algorithm 2 Algorithm for SAM (Score Allocation Model)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Input: K (Number of topics): Integer
Input: BN (Number of burn-in iteratoins): Integer
Input: N N (Number of sampling iteratoins): Integer
Input: Y (Score Index): A D × J matrix, Y [d, j] ∈ {1, · · · , Qj }, ∀d = 1, · · · , D
Input: C (Topic Allocation Initial Value): A D-dimensional vector, C[d] ∈ {1, · · · , K}, ∀d = 1, · · · , D
Input: γ: Hyperparameter
Output: CC: A D × K matrix
Initialisation:
Define: CC is a D × K matrix.
Define: N kj is a K × J matrix, N kj1 is a K × J matrix, N kj0 is a K × J matrix
For d ← 1 to D do:
For j ← 1 to J do:q ← Y [d, j], k ← C[d]N kj[k, j] ← N kv[k, j] + Qj ,N kj1[k, j] ← N kj1[k, j] + q, N kj0[k, j] ← N kj0[d, k] + Qj − q

32

16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

End forEnd forCollapsed Gibbs Sampling Loop:define: p1, p0, pA, p is a K-dimensional vector.
For nn ← 1 to BN + N N do:
For d ← 1 to D do:
For j ← 1 to J do:q ← Y [d, j], k ← C[d]N kj[k, j] ← N kv[k, j] − Qj ,N kj1[k, j] ← N kj1[k, j] − q, N kj0[k, j] ← N kj0[d, k] − (Qj − q)
End forFor k ← 1 to K do:
For j ← 1 to J do:
For q ← 0 to Qj do:
If q ≤ Y [d, j] do:
p1[k] ← p1[k] ∗ (N kj1[k, j] + q + γ)
End IfIf q > Y [d, j] do:
p0[k] ← p2[k] ∗ (N kj0[k, j] + Qj − q + γ)
End IfpA[k] ← pA[k] ∗ (N kj[k, j] + q + 2 ∗ γ)
End forEnd forp[k] ← p1[k]∗p2[k]pA[k]End fork ∼ Cat(p)For j ← 1 to J do:q ← Y [d, j]
N kj[k, j] ← N kv[k, j] − Qj ,N kj1[k, j] ← N kj1[k, j] − q, N kj0[k, j] ← N kj0[d, k] − (Qj − q)
End forIf nn > BN :CC[d, k] ← CC[d, k] + 1
End forEnd forEnd for

B.3

Pseudocode for TSAM (Text-Score Allocation Model)

Algorithm 3 Algorithm for SAM (Score Allocation Model)
1
2
3
4
5
6
7
8
9
10
11
12
13

Input: K (Number of topics): Integer
Input: BN (Number of burn-in iteratoins): Integer
Input: N N (Number of sampling iteratoins): Integer
Input: X (Document Index): A N -dimensional vector, X[i] ∈ {1, · · · , D}, ∀i = 1, · · · , N
Input: Y (Score Index): A D × J matrix, Y [d, j] ∈ {1, · · · , Qj }, ∀d = 1, · · · , D
Input: W (Word Index): A N -dimensional vector, W [i] ∈ {1, · · · , V }, ∀i = 1, · · · , N
Input: Z (Topic Allocation Initial Value): A N -dimensional vector, Z[i] ∈ {1, · · · , K}, ∀i = 1, · · · , N
Input: C (Topic Allocation Initial Value): A D-dimensional vector, C[d] ∈ {1, · · · , K}, ∀d = 1, · · · , D
Input: α, β, γ: Hyperparameters
Output: ZZ: A N × K matrix
Output: CC: A D × K matrix
Initialisation:
define: ZZ is a N N × N matrix.

33

14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

Define: CC is a D × K matrix.
define: N kv is a K × V matrix, N k is a K-dimensional vector, N dk is a D × K matrix.
Define: N kj is a K × J matrix, N kj1 is a K × J matrix, N kj0 is a K × J matrix
For i ← 1 to N do:v ← W [i], k ← Z[i], d ← X[i]N kv[k, v] ← N kv[k, v] + 1, N k[k] ← N k[k] + 1, N dk[d, k] ← N dk[d, k] + 1
End forFor d ← 1 to D do:
For j ← 1 to J do:q ← Y [d, j], k ← C[d]N kj[k, j] ← N kv[k, j] + Qj ,N kj1[k, j] ← N kj1[k, j] + q, N kj0[k, j] ← N kj0[d, k] + Qj − q
End forEnd forCollapsed Gibbs Sampling Loop:define: p1, p0, pA, p is a K-dimensional vector.
For nn ← 1 to BN + N N do:
Sampling Z:
For i ← 1 to N do:v ← W [i], k ← Z[i], d ← X[i]N kv[k, v] ← N kv[k, v] − 1, N k[k] ← N k[k] − 1, N dk[d, k] ← N dk[d, k] − 1
For k ← 1 to K do:
kv[k,v]+β(N dk[d, k] + C[d, k] + α)p[k] ← N
N k[k]+V ∗β
End fork ∼ Cat (p)N kv[k, v] ← N kv[k, v] + 1, N k[k] ← N k[k] + 1, N dk[d, k] ← N dk[d, k] + 1
If nn > BN :ZZ[i, k] ← ZZ[i, k] + 1
End forEnd forSampling C:
p1 ← 0K , p0 ← 0K , pA ← 0K
For d ← 1 to D do:
For j ← 1 to J do:q ← Y [d, j], k ← C[d]N kj[k, j] ← N kv[k, j] − Qj ,N kj1[k, j] ← N kj1[k, j] − q, N kj0[k, j] ← N kj0[d, k] − (Qj − q)
End forFor k ← 1 to K do:
For j ← 1 to J do:
For q ← 0 to Qj do:
If q ≤ Y [d, j] do:
p1[k] ← p1[k] ∗ (N kj1[k, j] + q + γ)
End IfIf q > Y [d, j] do:
p0[k] ← p2[k] ∗ (N kj0[k, j] + Qj − q + γ)
End IfpA[k] ← pA[k] ∗ (N kj[k, j] + q + 2 ∗ γ)
End forEnd forp[k] ← p1[k]∗p2[k](N dk[k] + α)pA[k]End for

34

70
71
72
73
74
75
76
77
78
79
80

k ∼ Cat(p)For j ← 1 to J do:q ← Y [d, j]
N kj[k, j] ← N kv[k, j] − Qj ,N kj1[k, j] ← N kj1[k, j] − q, N kj0[k, j] ← N kj0[d, k] − (Qj − q)
End forIf nn > BN :CC[d, k] ← CC[d, k] + 1
End forEnd forEnd for35