Discussion Paper No. 66

Measuring Large-Scale Market Responses from Aggregated Sales- Regression Model for High-Dimensional Sparse Data Nobuhiko TeruiandYinxing Li

February, 2017

Data Science and Service Research
Discussion Paper

Center for Data Science and Service Research
Graduate School of Economic and Management
Tohoku University27-1 Kawauchi, Aobaku
Sendai 980-8576, JAPANMeasuring Large-Scale Market Responses from Aggregated Sales- Regression Model for High-Dimensional Sparse Data Nobuhiko Terui1
andYinxing Li

February, 20171

Terui acknowledges a grant by JSPS KAKENHI Grant Number (A)25245054.
Tohoku University, Graduate School of Economics and Management, Kawauchi Aoba-ku,
Sendai, 980-8576, Japan; terui@econ.tohoku.ac.jp
1Measuring Large-Scale Market Responses from Aggregated Sales- Regression Model for High-Dimensional Sparse Data –Abstract
In this article, we propose a regression model for high-dimensional sparse data from storelevel aggregated POS systems. The modeling procedure comprises two sub-models—topic
model and hierarchical factor regression model—that are applied sequentially not only foraccommodating high dimensionality and sparseness but also for managerial interpretation.
First, the topic model is applied unusually to aggregated data to decompose the daily
aggregated sales volume of a product into sub-sales for several topics by allocating each unit
sale (“word” in text analysis) in a day (“document”) into one of topics based on joint purchase
information. This stage reduces the dimension of data inside topics because topic
distribution is not uniform and product sales are allocated mostly into smaller numbers of
topics. Next, the market response regression model in the topic is estimated by using
information about other items in the same topic. That is, we construct a topic-wise market
response function by using explanatory variables not only of itself but also of other items
belonging to the same topic. Additional reduction of dimensionality remains necessary foreach topic, and we propose a hierarchical factor regression model based on canonicalcorrelation analysis for original high-dimensional sample spaces. Then we discuss feature
selection based on a credible interval of parameters’ posterior density.
Empirical study shows that (i) our model has the advantage of managerial implications
obtained from topic-hierarchical factor regression defined according to their contexts, and (ii)
it offers better fit than does conventional category regressions in-sample as well as out ofsample.
Keywords: Topic Model, Hierarchical Factor Regression, Dimension Reduction, High
Dimensional Sparse Data, Feature Selection

2

1. Introduction
Disaggregated store data of scanner panel records have been analyzed using many models
and from many different perspectives.

For example, many choice models have been

proposed based on the theories of microeconomics and consumer behavior to understandcustomers and explore the effectiveness of the strategies suggested by said models.
The active use of daily aggregated store data—POS data that are accumulated
automatically at customer check out points—is important for most merchandisers, even for
those without a membership system.

Most traditional methods of analyzing POS data

specify market response function after the range of products is limited to a specific category
and the number of product is smaller than the number of records (days).

This category-based approach is useful when applied to products from well-recognized categories dependingon the validity of the assumed categories.

However, it cannot be applied to all products in astore, particularly to products that are purchased infrequently over observational periods.

It

is possible to lose useful information from store data when using the category-based
approach.
By contrast, for this purpose, it is well known that scanning entire databases creates room
for possibly discovering unexpected hidden patterns of joint purchases, which would bringnew insights for marketing management by understanding the market baskets or shopping
contexts of their customers.

The POS data in a store contain records of the number of sales,

prices, and promotions for 8,000 products.

The direct use of these variables as covariates to

explain sales is prohibited because the data is sparse and includes many zeros, and the size ofthe covariate matrix in the market response function is intractably large, that is, the entire dataof a POS is Big Data.

Even when the model can be estimated, overfitting occurs due to theso-called “N < P” problem, where N and P, respectively, refer to the numbers of samples andcovariates.

Thus, we need to generate smaller-sized datasets in some ways, for example, bydecomposing larger ones into several datasets or reducing the dimension of the data matrix.
In this study, we relax these restrictions and do not assume category by applying entire
products.

The proposed model is composed of two sub-models.

The first sub-model uses

the topic model to reduce the dimension of the original data space by decomposing it into aprespecified number of sub-datasets and uncovers a hidden structure by aggregating3

individual product purchases with store POS data, to which end we apply the disaggregateddata model (topic model) to aggregated data.

The second module solves the N<P problem

by reducing the dimension of the covariate space, to which end we propose a hierarchicalfactor regression model interpretable as a Bayesian canonical correlation model to estimate
the market structure in a lower-dimensional space between the dependent variable and thecovariates, where market response functions are estimated regularly given that there are notmany zeros in the reduced dimensional space.

Finally, the market structure in the high-dimensional original data space is recovered by converting the estimated structure in a
reduced-dimensional space to the original space.

An overview of the proposed model isshown in Figure 1.

Figure 1: Overview of ModelIn the next section, we apply the topic submodel to aggregated sales data and generate
sub-datasets, each of which contain purchases pertaining to each topic.
shopping context in our study.

We interpret topic as

These datasets already have smaller dimensions compared tothose of the original data space because it is likely that products sales are not allocated toevery topic evenly.
In Section 3, conditional on the datasets for topics, we use the second submodel toreduce dimensionality further to ensure that it is feasible to estimate topic-wise market
response functions for individual topics. More specifically, topic-wise market response
functions for products are estimated by using covariates related to variables of the productsthat belong to a given topic.

This market response structure is estimated in a reduced-

dimensional space and is then converted to the original space to extract sets of operational
covariates that affect sales.
data.

In Section 4, we report an empirical study performed using POS

The concluding remarks are given in Section 5.

2. Dimension Reduction Using Topic Model
2.1. Decomposing Aggregated Sales into Sub-sales by Shopping Context
Consumers have reasons when they purchase products in their shopping trips. Their
4

motivations, in other words, contexts, are buried in aggregated sales.

For example, out of

50 sales of a chocolate, 15 could be consumers purchasing for themselves, 25 for giftingpurchased jointly with a card, and 10 for cooking purchased jointly with flour. The
decomposition of total sales into several contexts, that is, topics, leads to better
understanding of the market and helps design efficient marketing strategies by targeting subsales in various contexts and conducting marketing to meet heterogeneous needs in thecorresponding topics.

We employ the topic model, which has been applied successfully totext analysis of natural language processing, for accommodating the latent topics inaggregated sales data.
The topic model is a reduced-dimensional model that has been applied successfully to text
analysis for determining the frequency of “words” in “documents.”

We employ the latent

Dirichlet allocation (LDA) model developed by Blei et al. (2003), which is well-establishedin natural language processing and used in a variety of disciplines.

The LDA model is a

generative model that allows for sets of observations to be explained by unobserved groups,
explaining why some parts of the data are similar.

It is based on the assumption that eachdocument can be viewed as a mixture of various latent topics, where the topics follow a
multinomial distribution over words.

|

denote the i-th word in document d andthe latent topic of the i-th word in document d.

|vocabulary (v) distribution of
(

Let

|

~

document d.

|

) and

|
|

The model assumes that thein topic k follows a multinomial distribution

follows a multinomial distribution (

|

~

) in

Then, the model describes the probability that vocabulary v appears in

document d and is represented as the sum of the products of topic distribution andvocabulary distribution in K possible ways:
K

K

k 1

k 1

p  v | d    p  v | k  p  k | d    v|k k |d .
The most common method of estimating the parameters

|

and

(2)
|

is Bayesianinference by using data augmentation of latent parameter z by way of full conditional
posterior density to evaluate the posterior density of these parameters.

When there is alarge volume of text data like, as our study, Gibbs sampling of the parameters requires a5considerable amount of time.

Therefore, we employ a collapsed Gibbs sampling, whichanalytically uses the natural conjugate of the prior distribution to integrate{| ,1, … , } is the vocabulary distribution in topic k, and {

|

,

|

and

1, … ,| .

is thetopic distribution in document d.
We first apply the topic model to POS data so that one unit of an item purchased on each
day in a store is allocated to one of the topics (segments) based on information about jointpurchase with other items.

That is, the number of sales of each product at time t isdecomposed into several topics, and datasets are constructed for topic-wise market response
functions.
In this study, one unit of an item purchased in each day in a store is allocated to one of the
topics (segments) based on information about joint purchase with other products.

The

vocabulary v of text analysis corresponds to a product “j” and the document d means the
day t in our study. Let Yjt denote the number of sales of product “j” on day “t” andYt  Y1t ,Y2t ,...,Ynt ' denote the vector of the total number of sales on the same day. In thecontext of text analysis, Yt corresponds to the frequency vector of nt types ofvocabularies.

The frequency of Y jt of the j-th vocabulary is allocated to each topic

proportional to the probability that each word from that vocabulary, i.e., unit sale of j-th
product, belongs to each topic p  j | t  

K

K

k 1

k 1

 p  j | k  p  k | t    j ,kk ,t . Based on the

allocation probability  j , k k ,t , we divide Y jt into sub-sales in topic k, denoted by Y jt( k ) ,according to the allocation

Y jt( k )  Y jt  E  j ,k k ,t 

(3)so that the aggregated sales are represented by the sum of topic-based sub-sales
K

Y jt   Y jt( k )k 1

2.2. Topic-wise Market Response Function
We have “k Bags” of product sales according to topics and we build market response
functions for the respective topics.

Contrary to the usual market response functions, we
6(4)

incorporate variables pertaining to other products in the same topic in the form of covariates
because the topics are extracted from joint purchase information with other products.
we define the market response function of product item j in topic k, Y jt

(k )

Then,

, not only by its

marketing variables such as price and promotions, but also other items’ sales and theirmarketing variables:

Y jt ( k )   0( kj )  α (jk ) ' X jt    m( k )Ymt ( k )   γ (mk ) ' X mt ( k )   jt ( k ) ,
m j

m j

where X jt is the vector of the marketing variables of item j,t  1,..., T j ( k ) (5)Ymt ( k ) is the number of salesof different items ( m  j )allocated similarly to topic k, and Xmt(k )is the vector of theirmarketing mix variables.
The forecast is defined by the sum of predictors constructed in each topic:Yˆit  Yˆit (1)  Yˆit (2)    Yˆit ( K )(6)This step reduces the dimensions of the variable because the distribution of allocation



probability  j ,k k ,t

 is usually not uniform but is concentrated on a few topics.

3. Dimension-Reduced Model for High-dimensional Market Responses
3.1 Hierarchical Factor Regression
Next, we consider the regression model for each topic between high-dimensional variables

Y ( k ) and X ( k ) in topic k. We delete the superscript (k) for ease of reading hereafter.
Now, in case of the multivariate regression model for Py -dimensional Y and Px dimensional X

Y  FX  e ,(7)

we consider the situation that the structural coefficient matrix F cannot be estimateddirectly due to high dimensionality of the original variable spaces.
model for the high-dimensional variables of Y and X .

We define a class ofFirst, we assume that they,

respectively, have dimension-reduction models for defining their marginal distributions7

p  Y | U, a  and p  X | V, b 

Yi  Uai  η yi ,Xi  Vbi  ηxi ,i  1,..., N(8)i  1,..., N ,(9)f y   Py  -dimensional vector and U is a Py  f y matrix, and bi iswhere a i is af x   Px  -dimensional vector and V is a Px  f x matrix.

By stacking the vectors withrespect to i to generate matrix forms, we haveY  Ua  η y(10)

X  Vb  η x ,(11)

where the augmented vectors are Y : Py  N , a : f y  N , η y : f y  N , X : Px  N , b :

f x  N , and η y : f y  N , and we assume that η y～N 0,  y and η ～N  0,   .
x

x

As for the joint distribution of Y and X , we assume that they are conditionallyindependent of the common parameter H.

That is,p  Y, X | U, a, V, b   p  Y | U, a, H  p  X | V, b, H  p  H | a, b  .

(12)More specifically, in the structure equation (1) between Y and X in the original space, e isthe error with zero mean and assumedly independent of X .
Instead of dealing with (7), following Brynjarsdotter and Berliner (2014), we assume that there
is a relationship in the reduced-dimensional space in terms of the hierarchical multivariate
regression model:a  Hb  ε(13)where the regression coefficient matrix H is of the order f y  f x and the error matrix ε isof the order f y  N whose columns are assumed to independently follow N 0,  I .
2This reduced-dimensional space is called “crystallized space” in Brynjarsdotter andBerliner (2014).

The motivation of the model is that the projection of X onto Y is

infeasible, and direct inference on (7) is not applicable. They estimated H by usingˆ  UHbˆ ˆ  UHVˆ ' X under the restriction ofrelationship (13) for forecasting Y as Y
8orthogonality on V in spatio-temporal modeling, but the estimation of structure F was outof the scope of their study.

This model assumes the presence of a relationship H between

Y and X , and they are independent conditionally of H . This model can be interpreted asa Bayesian multivariate canonical correlation model.
In the above, H plays the intermediate roles of a and b in the form of regression, and

Y and X are independent conditionally of H . That is, by considering that a and b
are interpreted as the principal components or factors of Y and X , they are determined byusing their relations by way of H .
3.2 Recovering Structure in High Dimensional Space
When X is given, structural equation (7) defines the conditional distribution.

Under

the assumption of conditional expectation of error term as E| x e  0 , the conditional
probability measure of Y on X yields

E| x  Y   FX  E| x e   FX

(14)Then, by taking expectation with respect to the probability measure of X ,Ex  E|x  Y   FEx  Xi.e. μ y  Fμ x



(15)



In terms of variables in the reduced-dimensional space, Ex E|x  Y   E  Y   Ua andE x  X   Vb , (15) induces the following relationshipUa  FVb .

(16)

In turn, the matrix F of the structure connecting the original variables in high-dimensional
sample space is obtained by

F  Uab ' V '  Vbb ' V '
The

joint

prior

densityoffactormodel

1parameters(17)p  U | a,  y  p   y  ,p  V | b,  x  p   x  , p  a |  a  p   a  , and p  b |  b  p  b  are also specified as normalinverted gamma conjugate priors for Gibbs sampling, as is applied to standard factor models,e.g., by Lee (2007).

Then, the posterior distributions ofp  U, a | H , Y andp  V, b | H, X  are derived by using the procedure of Bayesian factor models. Under the9assumption of normal-inverted gamma prior distribution forp  H | h  p  h  ,p  H | a, b,  2  is also available analytically, as is the posterior density of coefficientparameters in the normal linear regression model.
Then, the joint posterior density of all types of parameters in models (7)–(17) isrepresented byp  U, a, V, b, H,  2 , F,  y ,  x ,  h | Y, X  
p  U, a | H, Y  p  V, b | H, X  p  H | a, b,  2  p  F | U, a, V, b  p  2 | a, b, H  p  U | a,  y  p   y  p  V | b,  x  p   x (18)

 p  a |  a  p   a  p  b |  b  p   b  p  H |  h  p   h  p  2 
We note that p  F | U, a, V, b  is a degenerated density restricted as   U, a, V, b, F   0
in (17).

In order to recover F in the original high-dimensional space, we use the marginal

posterior density p  F | Y, X  , which is marginalized numerically in terms of the MCMC

procedure given by (18).

The steps of the MCMC procedure are given in the appendix.

We propose a doubly reduced dimensional regression model by applying sequentially the
topic model first and the hierarchical factor regression model thereafter.

We call this topic-

hierarchical factor regression in the following.
4. Empirical Application
4.1 Data
We applied the model to daily POS (point of sales) data of a store recorded between May
6, 2002, and May 6, 2003.

The dataset contains information about 7,912 items over 363

days, and a total 3,720,419 purchases were recorded.

POS data contain daily numbers ofsale for purchased items and their price, and three types of marketing promotional variables,
namely, two types of display and a feature.

There are no records of marketing variables foritems that were not purchased.
4.2 Topic Extraction
When we applied the first topic model, we set the number of topics k = 10.

The LDAmodel was estimated by using the collapsed Gibbs sampler under the conjugate Dirichlet prior10

distributions of hyper parametersandfor topic distribution and vocabulary distribution,

,…,

, and

respectively.
~

Following Griffiths and Steyvers (2004), we set~= 50/k and,…,0.1 for every element of

their vectors.
The results of the topic model are given in Table 1 and shown in Figure 1. Table 1 lists theitems with the top 20 highest probabilities, ˆj , k for k = 1, …, 10, where the item is denotedas the product number in braces and the category name is shown. For example, the first itemin Topic 1, “[6453]milk” indicates a product with identification number 6453 in the productitems.

Table 1: Categories and Items in Topics

4.3 Topic-Hierarchical Factor Regression for a Milk ProductWe applied the model to the sale of a brand of milk (JANcode:4902705065161), denoted

Y , for developing the market response model. Thus, we consider the regression model witha univariate dependent variable.

Y contains 21,482 total sales in 344 days. The first 324

days were used for estimation and last 20 days were used to validate the estimate.

Figure 1: Sales of a Milk Product (JAN: 4902705065161)
Figure 1 shows the time series plot of Y .

The product is sold regularly with significant

spikes on some days, which could be caused by store promotional activity, and the level of sales
remains high over the summer season from July (60th day) to August (70th day).

Figure 2: Averaged Topic Distribution for a ProductFigure 2 shows the topic distribution of a product averaged over entire days, specificallydefined by11ˆj ,kˆk | 
where N = 344 and K = 10.

1 N ˆ ˆ  j ,kk|t , k  1,..., K ,N t 1(19)

We can observe that the topics have nearly equal weights,

except for the most frequent topic at k = 4 and the least frequent topic at k = 8.

Figure 3: Topic Sales Decomposition(k )Figure 3 shows the time series of topic-wise sales, Y jt , k  1,...,10 , which were generated

by using the averaged topic distributions.

The time series patterns exhibit seasonality withranges from one to three months, although a few of the ranges contain overlapping periods.
Topic 8 is rather exceptional with regularly sold pattern of relatively small numbers.

We notethat the allocated numbers are not integers in general according to formula (3).
To be specific, Topic 1 refers to the sales between September and October, and the sales
between January and March are classified in Topic 2.

Topic 4 refers to the sales in the summer

vacation period between July and August and the period between January and February, andTopic 10 refers to the sales at the start of the fiscal year in April.

The sales in summer andearly autumn are covered by Topic 7, and so on.
Combined time series plots of topics with averaged topic distribution and LDA topic model
decompose the sales over the entire year into clearly distinguished submarkets defined by theseasons, and among all, summer is most important season for the target product as mostly fourtimes sales as other topics.
4.4 Model Comparison
In empirical analysis, we set a univariate Y.

We then defined the number of reduced-

dimensional space of X as f x  20 for every topic.

According to the topic-hierarchicalfactor regression model, the estimated dependent variable in topic k is calculated by evaluating
the posterior means of related parameters and covariates1Yˆt ( k )  E  F ( k ) Xt ( k )   E U ( k )a ( k )b ( k ) ' V ( k ) '  V ( k )b ( k )b ( k ) ' V ( k ) '  X ( k )t12

 (17) 10 ˆ ( k )Yt | Data  .
 k 1and the posterior density of forecast is defined as p Yˆt | Data  p 

In the above, the covariate X t ( k ) in topic k contains four marketing variables, in addition to
the joint sales of other products and their marketing variables.
The model fit is evaluated by the root mean square error RMSE  Y  E Yˆ   / S
S

s 1

2

s

s

  is used as a point

for in-sample and out-of-sample data, where the posterior mean E Yˆsforecast.

Table 2 shows the RMSEs of in-sample and out-of-sample forecasts for various

models.
The first model is topic-hierarchical factor regression.

The structural regression model

defined by (7) in the original space contains 39,560 parameters, obtained as 7,912 item numbers
multiplied by five types of covariates, that is, the target item’s marketing variables and the saleof jointly purchased other items and their marketing variables.

Table 2: Model Fit and Comparisons
The model’s performance can be improved by extracting effective covariates to reduce thenumber of model parameters and increase the degrees of freedom.

Then, the second proposedmodel contains a reduced number of covariates after eliminating covariates with insignificant
parameter estimates.

We evaluated the posterior density p  F | Data  of the regression

coefficient, and we can apply the 95% credible interval test for determining the significance ofcoefficients to increase the number degrees of freedom.

Specifically, we selected covariateswith estimated coefficient parameters based on the criterion that 95% of the central region ofposterior density does not include zero.
Table 2 shows the total number of effective parameters, and Table 3 shows the number ofcovariates for the respective models in the 10 topics.
effective variables.

In total, 2,131 parameters remained as

The number of parameters decreased dramatically to approximately 5.4%of the number of parameters in the original space.

The numbers of selected covariates in therespective topics were almost proportional to the volumes of data allocated to the topics.
13

Table 3: Number of Effective Covariates
Then, we re-estimate the model by eliminating insignificant variables following the sameprocedure as before.

We call this model effective topic-hierarchical factor regression.

The third alternative model uses only the top 20 items with the highest topic probability ineach topic, called topic regression.

This selection was followed by studies on topic model

analysis in natural language processing, adopted usually in machine learning studies.

Theestimation process is the same, except the 2nd reduced-dimensional factor model is not used.
The fourth alternative model assumes a category a priori for the target item and uses only items
from the same category.
called category regression.

This model has been used conventionally in marketing, and it isThis is the benchmark model.

This category contains 48 items.

In addition to four marketing variables, the model contains 47 times 5 covariates of other items
sales and their marketing variables, so 239 is the total number of covariates.
Table 2 shows the results of model fit in terms of RMSE.

First, the conventionally usedcategory regression model performs the worst in in-sample and out-of-sample forecasts.

This

means that shopping contexts expressed by the topics are likely to have important roles, and thenarrowly defined category-based analysis loses useful information hidden in purchase records.
The comparison of (i) topic-factor and (iii) topic regression models indicates rather mixed
results in the in-sample and the out-of–sample forecasts. Although both models use informationof all items when topics are extracted, (ii) uses only 20 variables in regression, and it might lose
information in forecasting, even though it is adequate to explain in-sample data.
Finally, the comparison of (i) the topic-factor and (ii) the efficient topic-factor models
clearly supports (ii), in particular, for out-of-sample forecasting.

Figure 4:Model Fit and Forecasting

4.5 Efficient Topic-Factor Reduced Dimensional Regression
Model (ii) is supported as the best, and thus, we report the estimation results of that modelin greater detail in this section.
14

Figure 5: Estimated Sales in Topics
Table 4(1), (2): Feature Selection for Topics

Figure 5 shows the in-sample fit for the respective topics.

In addition to fit for total salein Figure 4, we observe that the model performs well for every topic.

Table 4(1) and (2)indicate the results of feature selection in terms of 95% credible interval for recovered posterior
density.

The significant estimates of “sales”, “price”, “promotion 1,” and “promotion 2” aretabulated up to ten in the order of absolute magnitude.

The estimated coefficients are given

after the name of category and product ID number in parentheses.

The coefficient means

elasticity for log-transformed sales and price, and it holds approximately for other binary
promotional variables.
(i)

We observed the following:

The sales variables of other products are identified as effective covariates acrosstopics.

This is reasonable because we modeled the regression by using jointpurchase patterns.
(ii)“price” variables are chosen as effective in topic 6 and in many cases in topic 10.
of them are cross elastic, and self-elasticity is not significant.
targeted product does not change much.

All

The price of thisConventional category regression shares

this result for self-elasticity.
(iii)“promotions” are effective for topics 5, 6, 7, 9, and 10.

(iv)

Topic 9 has the most number of effective covariates.

The sales of foodstuffs are

selected as features, and we interpret this fact as milk is purchased not for drinking
but for cooking.
(v)

Promotions of chocolate and other drinks are effective.

In topic 4, which has the most volume, the sales of water and juice are synchronizedwith the sale of the target product and are opposite to the sales of chocolates andcookies from June to August.

This may be consistent with the seasonal pattern ofsummer from June to August.

15

Table4(1): Feature Selection for Topics

Table 4(2): Feature Selection for Topics

5. Concluding Remarks
The volume of high-dimensional data is growing with progress in network technology.
These high-dimensional data contain many zeros, that is, sparse in data spaces, as is the casewith POS data in our study.

That is, the daily number of product items purchased isconsiderably smaller than that of items displayed in a store, and many items are recorded ashaving zero sales and information about their price and promotional variables are notrecorded, thus producing yet another data entry with zero value.

POS data are used to findeffective relationships between product items and promotional variables to facilitate efficientmanagement.

In these cases, statistical models that accommodating simply high-dimensional data do not work.
We proposed a regression model for high-dimensional and sparse data.

In the proposedmodel, two types of dimension-reduction models, namely, topic model and canonicalcorrelation model, are used sequentially.

More specifically, the store-level aggregated sales

and marketing mix data are analyzed using a model that combines the topic model with theregression model.

First, the topic model decomposes sales of product items into several

topics by allocating each unit sale (“word” in text analysis) in a day (“document”) into one of
topics based on joint purchase information.
Next, conditional to the topic model estimates, the market response function of an item isestimated by using information regarding variables on the items inside a topic.

That is, weconstruct topic-wise market response functions of an item by using explanatory variables not
only of said item but also of the other items belonging to the same topic.

In addition, theexplanatory variables include not only marketing mix variables but also the numbers of salesof product items in the same topic.

We can expect unusual findings from these response

functions because sets of related items are not considered in advance and they are notobtained by conventional category-based market response functions.
Finally, we forecast the sales of an item by summing up respective predictors constructed
16

for topics.

We compared the model performance with that of conventionally predeterminedcategory regression model to show that (i) our model has the advantage of offering
managerial implications obtained from topic-wise regressions defined according to theircontexts, and (ii) it has better fit than does the category regression model.
We have basically kept the number of topics for topic model and the numbers of factors
for hierarchical factor regression model as fixed in the empirical application, mostly due tothe restrictions on computational times.

By using R under usual PC environment, it took

four days for estimating topic model, and two days for the part of hierarchical factor
regression model when numerically calculating inverse of large scale matrix to recover the
structure in original space.

We leave this model selection problem for future research bydeveloping more efficient computation procedures, although we currently recognize that theestimation procedure of second submodel’s part could be eased if we employ parallel
computing for topic wise market response functions.

17

Reference
Blei, D.M., Ng A.Y. and Jordan, M.I.(2003) "Latent Dirichlet allocation," Journal of Machine
Learning Research, 3, 993-1022.
Blei, D. M. (2012), "Introduction to Probabilistic Topic models," Communications of theACM,55, 77-84.
Brynjarsdottir, J. and Berliner, L.K. (2014), “Dimension-reduced modeling of spatio-temporalprocess,” Journal of American Statistical Association, vol. 109, 1647-1659.
Iwata,T. and Sawada H.(2013), "Topic model for analyzing purchase data with priceinformation," Data Mining and Knowledge Discovery, 26, 559-573.
Lee, S. Y. (2007), Structural Equation Modeling: A Bayesian Approach, Wiley, New York.
Hanssen, D.M., L.J. Parsons and R.L. Shultz (2001), Market Response Models: Econometricand Time Series Analysis, 2nd ed., Boston, MA: Kluwer Academic Press Inc.
Rossi, P.E., G. Allenby and R. McCulloch(2005), Bayesian Statistics and Marketing, Wiley,
New York."

18

Appendix: MCMC Algorithm
1. LDA Topic Model
Under

the

prior

~

α, β

distributionsof

1, … ,~, andDirichlet

distributions
1, … ,

onhyperparameter, where M is the number of

documents and K is the number of topics. Following Griffiths and Steyvers (2004), we set thevectors with α = 50/K and= 0.1. Then, we used the collapsed Gibbs sampler to obtain

posterior density in the closed form asp  zd ,i  k | wd ,i  v, w \ d ,i , z \ d ,i , α, β  nk\ d,v,i   v

nd\ d, k,i   k

nk\ d, ,i    v ' nd\ d ,i    k '
v'

,

k'

where wd ,i means word i in document d, zd ,i is the latent topic of word i in document d,

 d means topic distribution of document d, and k is the vocabulary distribution of topic
k. w \ d ,i is the set of all words from the text data except word
except, ,, ,z \ d ,i is the set of all topics\ d ,iis the number of words in document d, nk ,v refers to the frequency of\ d ,iword v in topic k except for that of word i in document d, and nk , n\ d ,ik ,v .

v

2. Hierarchical Factor Regression Model
2.1 Prior Density
Prior

Settinga k ~ N μ 0 , Φ 0  , b k ~ N μ 0 , Φ 0 

μ 0 ~ 0 K 1 , Φ 0  I K K  100

Y ~ N a 0 , Σ y  X ~ N b 0 , Σ x ,

a 0  0, b 0  0

Σ y ~ IGr0，s 0  Σ x ~ IG r0，s 0 

r0  2, s 0  2,

a ~ N μ 0，  0 Λ 0 12



μ 0  0 , Λ 0  I  100

Φx, Φy ~ IWq R 0 ' ,  0 

R 0  I ,  0  100

 0 2 ~ IG 0， 0 

 0  2,  0  2

19

2.2 Conditional Posterior Distribution for MCMC
(1) ai | yi , U, Σ y , Φ ypai | yi , U, Σ y , Φ y   N Φ y1  U' Σ 'y U U' Σ U, Φ  U Σ U  
1

1
y

'
y

'

1

'
y

1

(2) U k | Y, a, Σ yk1
p(Uk | Y, a, Σyk) ~ N  a*k a*k ' a*k Yk*，Σ yk  a*k a*k '
1

1



(3) Σ ψκ | Y, a











 



p Σ yk | Y, a ~ IG n  rk  / 2, s 0  2 1 Yk* ' I n  a *k ' a *k a *k ' a *k Yk*
1(4) Φ y | Y, UpΦ y | Y, U   IWq aa' R 0 ' , n   0 (5) bi | xi , V , Σ x , Φ xpbi | xi , V, Σ x , Φ x   N Φ x1  V ' Σ 'x V

 V Σ V, Φ  V Σ V  
1

'

1
x

'
x

'

1
(6) Vk* | X, b, Σ xk1p(Vk* | X, b, Σxk) ~ N  b*k b*k ' b*k X*k，Σ xk  b*k b*k '
1

1



(7) Σ xk | X, b













 

p Σ *xk | X, b ~ IG n  rk  / 2,2 1 X *k ' I n  b *k ' b *k b *k ' b *k X *k1(8) Φ x | X, VpΦ x | X, V   IWq bb'R 0 ' , n   0 (9) H | a, b,  2







p H | a, b,  2  N b' b  Λ 0  Λ 0  0  b' a , b' b  Λ 0 1(10) Λ n | Λ 0

Λ n  b' b  Λ 0(11)  | ai , bi , H
2

20

1



'
x

1





n
1


p  2 | ai , bi , H  IG  n  ,  n  a' a  μ 0 ' Λ 0 μ 0  μ n ' Λ n μ n  ,
2
2


where









1
ˆ  b' b  Λ 1 Λ μ  b' a  Λ  b T b  Λ
μ n  b' b  Λ 0  Λ 0 μ 0  b' bH
0
0 0
0 ,
, n

n
2

 n   0  , and , β n  β 0 1
a' a  μ 0 ' Λ 0 μ 0  μ n ' Λ n μ n  .
221

Table 1: Categories and Items in Topics
Topic 1
Topic 2
Topic 3Topic 4
Topic 5[6453]milk[7262]milk[4912]milk[6453]milk[4808]milk[3630]milk[6455]milk[6453]milk[6615]water[4787]yogurt[6524]energy drink[3630]milk[3630]milk[4928]yogurt[7262]milk[6795]cola[6795]cola[4787]yogurt[3630]milk[3596]milk[3596]milk[3596]milk[6795]cola[6795]cola[6795]cola[6800]carbonated flavor [6800]carbonated flavor [3596]milk[3596]milk[3630]milk[4787]yogurt[5506]cup noodle[6800]carbonated flavor [6800]carbonated flavor [6800]carbonated flavor[4928]yogurt[5879]yogurt[6455]milk[4787]yogurt[4928]yogurt[6622]Janpanese tea and [4950]yogurt[4928]yogurt[4933]yogurt[3597]milk[463]yogurt[3835]cup noodle[463]yogurt[3496]cup noodle[4786]milk[6798]Chinese tea drink [463]yogurt[3712]cola[3729]Janpanese tea and[5506]cup noodle[3496]cup noodle[3833]cup noodle[5895]coffee drink[3238]water[6798]Chinese tea drink[453]yogurt[5887]yogurt[6798]Chinese tea drink [1846]coffee drink[4933]yogurt[2895]mayonnaise[6798]Chinese tea drink [3497]cup noodle[3706]cola[463]yogurt[2162]vegetable juice[5946]lactic acid bacteria[3496]cup noodle[6798]Chinese tea drink [6524]energy drink[6787]water[4997]yogurt[5946]lactic acid bacteri [3483]cup noodle[5946]lactic acid bacteria b[3706]cola[3735]sports drink[6833]coffee drink[463]yogurt[431]water[3497]cup noodle[3605]yogurt[3605]yogurt[6622]Janpanese tea and[1064]yogurt[5873]milk[6622]Janpanese tea and[3164]cup noodle[5946]lactic acid bacteria[3288]Janpanese tea and[5946]lactic acid bacteria [4915]milk[3319]sports drink[3497]cup noodle[3712]cola
Topic 6
Topic 7
Topic 8Topic 9Topic 10[7262]milk[6453]milk[408]cup noodle[4808]milk[6455]milk[6795]cola[3630]milk[411]cup noodle[3630]milk[3630]milk[3630]milk[3596]milk[3630]milk[7262]milk[7262]milk[6800]carbonated flavor [4787]yogurt[3635]milk[3596]milk[4808]milk[3596]milk[6795]cola[409]cup noodle[6795]cola[3596]milk[4928]yogurt[6800]carbonated flavor [3850]cup noodle[6800]carbonated flavor [4912]milk[4787]yogurt[4928]yogurt[5756]soy[3236]Chinese tea drink [6795]cola[3735]sports drink[3497]cup noodle[2895]mayonnaise[3735]sports drink[6800]carbonated flavor[6624]Janpanese tea and [463]yogurt[6746]salad oil[4787]yogurt[4787]yogurt[5887]yogurt[3496]cup noodle[3735]sports drink[5873]milk[6453]milk[4808]milk[6798]Chinese tea drink [3706]cola[463]yogurt[5946]lactic acid bacteria b[451]personal ice (cream) [5946]lactic acid bacteria[4243]soup[3712]cola[6822]carbonated flavor[453]yogurt[4912]milk[2397]lactic acid bevera [5879]yogurt[6557]energy drink[463]yogurt[4997]yogurt[6624]Janpanese tea an [3706]cola[4997]yogurt[3706]cola[4209]Janpanese tea and[3497]cup noodle[3734]sports drink[463]yogurt[5506]cup noodle[3503]cup noodle[2725]soy[5895]coffee drink[5887]yogurt[6787]water[3712]cola[5887]yogurt[6787]water[4928]yogurt[6622]Janpanese tea and [1064]yogurt[3496]cup noodle[6798]Chinese tea drink [6872]cola[450]personal ice (cream) [6833]coffee drink[4289]instant curry[5946]lactic acid bacteria[5895]coffee drink[5879]yogurt[3279]energy drink[4224]salad oil[5887]yogurt[5506]cup noodle

22

Table 2: Model Fit and Comparisons
Model

RMSE1(In Sample)

RMSE2(Out of Sample) Number of Covariates(a) Topic-Factor Dimension-Reduced

50.41

14.40339,560(200)(b) Effective T-F Dimension-Reduced

45.622

8.168

2131(200)(c) Topic Regression

46.61719.164

1000(d) Category Regression

93.47233.956

239

Table 3: Number of Effective Covariates
Topic:kNumber of
Variables

1

2

3

4

5

6

7

8

9

10

190

205

235

284

179

281

286

74

239

158

23

Table 4(1): Feature Selection for Topics
Topic

1

Topic

2

Topic

3

Topic

4

Topic

5

Sales
[4891]miso[4929]coffee drink[5025]rice cracker[5577]suger[3249]energy drink[7034]coffee drink[1927]family ice[3287]fruit juice[494]sauce[7384]premium ice

Coef
Price
2.318 [3064]rice cracker
2.184 [3523]cup noodle
2.141 [7211]barbecue sauce
2.093
1.946
1.942
-1.885
1.764
1.709
1.649
-

Coef
Prom1
0.079 [3064]rice cracker
0.030 [1639]cup noodle-0.001
-

Coef
Prom2
-0.019 [2092]tomato juice
0.010 [235]yogurt-

Coef-0.089-0.043
-

Prom3
-

Coef
-

Sales
[5790]biscuit and cookie[4919]family ice (cream)[3846]cup noodle[1345]chocolat[2727]soy[1826]dressing[89]coffee drink[3587]personal ice[6860]instant coffee[1413]biscuit and cookie

Coef
Price
3.000 [6012]cheese
1.948
1.764
1.764
1.680
1.674
1.532
1.462
-1.460-1.389
-

Coef
Prom1
-0.014 [2245]snack[2553]chocolat
-

Coef
Prom2
0.154 [5550]cup noodle-0.090 [3128]cup noodle[2112]instant soup-

Coef
-0.260-0.123
0.108
-

Prom3
-

Coef
-

Sales
[1287]cooked curry[4825]lactic acid bacteria b[1138]mayonnaise[494]sauce[6402]chocolat[3116]candy[469]dessert[166]personal ice (cream)[3559]cheese[883]carbonated flavor

Coef-2.0881.913-1.607
1.516
1.379
1.352
-1.316
1.294
1.272
1.128

Coef
-

Coef
Prom2
0.486 [2031]dressing[2030]dressing-

Coef-0.178
0.145
-

Prom3
-

Coef
-

Sales
[258]water[5304]biscuit and cookie[4309]instant curry[3731]carbonated flavor[4792]family ice (cream)[5184]chocolat[2334]snack[4094]fruit juice[6122]chocolat[6832]cola

Coef
Price
3.963 [3182]cup noodle-3.253
3.042
3.008
2.941
-2.626
-2.54
2.431
2.328
2.314
-

Coef
0.165 -

Coef
-

Coef
-0.049
-

Prom3
-

Coef
-

Sales
[7897]instant noodle[6815]black tea drink[3020]dressing[4297]cooked curry[6735]miso[6862][7717]energy drink[2486]candy[505]candy[6211]candy

Coef
Price
2.174 [5046]dessert-1.998 [4983]health food
1.781 [2297]snack
1.584
1.538
1.528
1.528
-1.443-1.413
1.410
-

Coef
Prom1
0.036 [4710]biscuit and cookie-0.015 [3419]snack
0.000 166]Principle of Chinese food
-

Price
-

Prom1
[3225]salad oil
Prom1
-

24

Prom2
[869]carbonated flavor
-

Coef
Prom2
0.451 [5255]chocolat
0.276 [5916]family ice-0.187 [3015]mayonnaise[6451]100% fruit juice
-

Coef
Prom3
-0.328 [2227]rice cracker-0.047-0.020-0.006-Coef-0.108-

Table 4(2): Feature Selection for Topics
Topic

6

Topic

7

Topic

8

Topic

9

Topic

10

Sales
[4124]coffee drink[7196]soup[3019]dressing[6540]100% fruit juice[713]yogurt[5884]vegetable juice[2493]candy[7158]family ice (cream)[6143]chocolat[1125]Principle of Chinese

Coef
Price
2.021 [7430]dessert-1.678 [1560]rice cracker-1.673 [3184]cup noodle-1.613 [7148]family ice (cream)
1.580
1.517
1.445
1.364
1.327
1.265
-

Coef
-0.2900.174-0.086
0.000
-

Sales
[1345]chocolat[2064]regular coffee[7166]yogurt[1264]instant curry[6269]soup[4097]rice cracker[6518]Janpanese tea andbarley tea drink[6103]chocolat[6428]yogurt[6122]chocolat

Coef
Price
2.085 [3631]dessert
1.754 [4686]chocolat
1.668
1.667
1.661
-1.464
-

Coef
Prom1
0.013 [1634]cup noodle
0.003 [1878]regular coffee- [3196]cup noodle[1142]Chinese food[3345]coffee drink[1273]cooked curry

1.429

-

-

-

-

-1.422
1.393
1.362

-

-

-

-

Sales
[3589]milk[6620]coffee drink[6772]suger[3493]cup noodle[2493]candy[5579]suger[3101]chocolat[359]personal ice (cream)[89]coffee drink[4786]milk

Coef-0.4890.424
0.232-0.2310.225-0.207
0.205
0.205
0.170
0.164

Price
-

Coef
-

Prom1
-

Coef
-

Sales
[1086]Principle of Chinese[4263]Principle of Chinese[1149]salad oil[1230]instant curry[6734]instant soup[4839]cheese[5988]dressing[274]candy[743]vegetable juice[1578]health food

Coef
1.415
1.291
1.193
1.185
-1.072
1.058
0.981
0.978
0.967
-0.947

Price
-

Coef
-

Prom1
[6891]rice cracker[6053]chocolat[2598]chocolat[1680]cup noodle[2534]chocolat-

Sales[4297]cooked curry[494]sauce[3287]fruit juice[7176]yogurt[3068]candy[3847]cup noodle[3010]dressing[6936]candy[7260]dessert[6831]carbonated flavor

Coef
Price
2.070 [476]dessert
1.979 [271]chocolat
1.578 [2659]chewing gum-1.340 [5007]personal ice (cream
1.283 [4407]barley tea
1.181 [6584]100% fruit juice-1.159 [1672]cup noodle
1.014 [1621]snack-0.960 [5021]yogurt
0.936 [6586]vegetable juice

-

Prom1
-

Coef
Prom1
-0.379 [4643]chocolat
0.332 [4407]barley tea
0.210-0.168
0.089
0.078
0.073
-0.069-0.055
0.053
-

25

Coef
-

Prom2
[3783]carbonated flavor
-

Coef
0.066
-

Prom3
-

Coef
-

Coef
-0.518-0.117-0.036-0.007
-

Prom3
-

Coef
-

-

-

-

-

-

-

-

-

Prom2
-

Coef
-

Prom3
-

Coef
-

Coef
Prom2
-0.965 [6316]energy drink-0.646 [6317]energy drink-0.532 [2167]vegetable juice-0.442 [2090]tomato juice-0.334 [2095]tomato juice[2156]vegetable juice[2959]dressing
-

Coef
0.470
-0.345-0.3130.143
0.088-0.054
0.010
-

Prom3
-

Coef
-

Coef
Prom2
0.121 [3895]cup noodle-0.012 [3394]dessert[5483]soup[3393]dessert
-

Coef
0.278
0.098
-0.073
0.012
-

Prom3
-

Coef
-

Coef
Prom2
1.035 [3994]dessert-0.532 [6777]suger
0.333 [6874]vegetable juice
0.272 [6781]bread crumb
0.171-0.105-

Figure 1: Overview of Model

26

Figure 1: Sales of a Milk Product (JAN: 4902705065161)

Figure 2: Averaged Topic Distribution for a Product27

Figure 3: Topic Sales Decomposition

28

Figure 4: Model Fit and Forecasting(a) Topic-Hierarchical Factor Regression(b) Efficient Topic-Hierarchical Factor Regression(c) Topic Regression(d) Category Regression

29

Figure5: Estimated Sales in Topics

30

31