Neurocomputing 424 (2021) 143–159

Contents lists available at ScienceDirect

Neurocomputing
journal homepage: www.elsevier.com/locate/neucom

Boosting prior knowledge in streaming variational Bayes q
Duc Anh Nguyen a, Van Linh Ngo b, Kim Anh Nguyen b, Canh Hao Nguyen a, Khoat Than b,⇑ab

Bioinformatics Center, Institute for Chemical Research, Kyoto University, Japan
School of Information & Communication Technology, Hanoi University of Science and Technology, No. 1, Dai Co Viet road, Hanoi, Viet Nama r t i c l e

i n f oArticle history:
Received 19 August 2019
Revised 12 August 2020
Accepted 10 October 2020
Available online 20 October 2020
Keywords:
Streaming variational Bayes
prior knowledge
online learning
short text
sparse datanoisy data streams

a b s t r a c t
Exploiting prior/human knowledge is an effective way to enhance Bayesian models, especially in cases ofsparse or noisy data, for which building an entirely new model is not always possible. There is a lack ofstudies on the effect of external prior knowledge in streaming environments, where the data comesequentially and infinitely. In this work, we show the problem of vanishing prior knowledge in streaming
variational Bayes. This is a serious drawback in various applications. We then develop a simple framework
to boost the external prior when learning a Bayesian model from data streams. By boosting, the priorknowledge can be maintained and efficiently exploited through each minibatch of streaming data. We
evaluate the performance of our framework in four scenarios: streaming in synthetic data, streaming sentiment analysis, streaming learning for latent Dirichlet allocation, and streaming text classification, incomparison with the methods that do not keep priors. From extensive experiments, we find that whenprovided good external knowledge, our framework can improve the performance of a Bayesian model,
often by a significant margin for noisy and short text streams.
Ó 2020 Elsevier B.V. All rights reserved.

1. Introduction
In the world of data explosion, designing a model that can capture continuously upcoming data is a basic need. By consideringthat new data depend on the past data in a probabilistic way, theBayesian approach is highly suitable to model data streams where
the data come sequentially and infinitely. Many researches havebeen successful in using this idea [1–5]. Through streaming learning methods, a model can be learned in a forward way without
revisiting the old data, and hence more efficiently in terms of timeand memory space. However, many challenges exist, including extreme sparsity and noisy data.
Sparsity in which the observed data are (extremely) sparse innature is prevalent in practice, such as user ratings in recommender systems [6,7], and posts/comments from social networks[8–10]. Sparse data contain little information, and thus pose a severe challenge for modeling even in cases of very large number ofsamples [11].1 Further, noisy data (e.g., comming from social netq
This research is funded by Vingroup Innovation Foundation (VINIF) annual
research support program in project code VINIF.2019.DA18, and by the Office ofNaval Research Global (ONRG), Air Force Office of Scientific Research (AFOSR), Asian
Office of Aerospace Research & Development (AOARD) under Award Number
N62909-18-1-2072.
⇑ Corresponding author.
E-mail address: khoattq@soict.hust.edu.vn (K. Than).
1
Although the proof of [11] focuses on a particular model, we believe that theirresult holds true for a large class of Bayesian models.

https://doi.org/10.1016/j.neucom.2020.10.026
0925-2312/Ó 2020 Elsevier B.V. All rights reserved.

works or online forums) with contradicting or irrelevant information
present inherent difficulties for modelling [12–16]. Fig. 1 shows two
examples where the learnt models seem to encounter severe overfitting as learning from noisy and sparse data. For those challenges,
designing an entirely new model is not always possible and possibly
takes a high cost. Therefore providing human knowledge is often agood choice to improve a Bayesian model.
Consider the task of learning from a data stream, where the data
come sequentially and infinitely. Existing learning methods havedifficulties to effectively exploit prior knowledge. Streaming variational Bayes (SVB) [1] can use the external knowledge as initialprior at the first step of the learning process. However, in othersteps, the prior is replaced by the posterior learned from the previous learning step. This strategy leads to losing information fromthe past, as shown in Section 3.2, and limits the effect of external
knowledge. Population variational Bayes (PVB) [2] employs a population distribution to capture streaming data. Masegosa et al. [3]develop SVB further to balance the old and new knowledge, alllearned from data, in a Bayesian way. Faraji et al. [5] study thesame problem as [3], while Theis and Hoffman [19] develop a variant of stochastic variational inference (SVI) [20] for data streams.
Due to the ignorance of external knowledge, those studies are limited when we want to exploit valuable knowledge in order to deal
with the three challenges above. Note that good prior knowledge
are now available in various forms including ontologies, open
knowledge bases (e.g., Wikipedia), word embeddings, unsupervised pre-trained models [21,22], Zipf’s law, etc.

D.A. Nguyen V.L. Ngo, K.A. Nguyen et al.

Neurocomputing 424 (2021) 143–159

For text data, by capturing the Zipf’s law [27,28], the performanceof topic models can be improved [29] in comparison with LDA [18].
In text classification, Lin et al. [30] used an asymmetric prior whichgave high weight for seed words of each class to gain better performance. In sentiment analysis, some studies [17,31] exploited aword list that consists of a predefined set of positive and negative
words in learning Bayesian models to determine the sentiment ofeach document. However, most of those existing researches focus
on static conditions, leaving an unexplored problem of how to
exploit human knowledge in streaming environments where thedata come sequentially and infinitely. Such an unexplored problem
is the object of interest in this work.
AA Another term related to prior is power prior [32,33]. Powerprior is used to balance the old knowledge learned from past data
and the new knowledge learned from the current minibatch ofdata. While the initial prior is gained even when there is no data
received, it can be provided from external knowledge that is outside of the observed data. Nonetheless, existing studies [5,3,19]
did not emphasize in the role of the external prior but concern
more on balancing the old and new knowledge which are alllearned from data.
Other related topic is stochastic methods for training a Bayesianmodel from a dataset of big size. Examples include stochastic variational inference (SVI) [20] and memorized online variational inference (MOVI) [34]. The problem setting in our work is learningfrom a data stream, where the data come sequentially and infinitely.
SVI and MOVI seem to be unsuitable for this problem setting. SVI not
only requires to know the data size in advance, but also assumes theability to have multiple learning passes over the whole data. MOVI
divides dataset into the fixed number of minibatches and iterates
through each minibatch multiple times. Both SVI and MOVI canwork well on a big (but fixed) training data. However, in the streaming environment, since the dataset size is infinite and the underlying
properties of the data may change over time, iteratively going
through all minibatches is intractable. Some works [19,2] alreadypointed out those limitations of SVI (and MOVI).

Fig. 1. Examples of severe overfitting when Streaming variational Bayes [1] does
learning from noisy and short text streams, where ’Batch size’ is the size of the data
received at each time step. (a) shows the accuracy of the Aspect and SentimentUnification model [17] for sentiment analysis. (b) shows the predictive probability of
the latent Dirichlet allocation model [18] for text analysis. Higher is better. Music
contains 136 K online reviews about musical instruments, collected from http://
www.cs.jhu.edu/~mdredze/datasets/sentiment/. Yahoo is the dataset consisting of
500 K questions, crawled from http://answers.yahoo.com/, each question is noisy
and extremely short [10]. The shadow shows the variance of the empirical result w.
r.t. different runs.

1.2. Roadmap

In this paper, we make the following contributions:

Section 2 briefly presents some preliminaries with basic notations in Table 1. Section 3 reviews the SVB framework and the
problem of vanishing priors. In Section 4, we propose the BPS
framework. Section 5 presents case studies in applying SVB and
BPS in streaming learning. Finally, Section 6 concludes our work.

 We show that SVB will quickly forget the prior knowledge
through the streaming learning process.
 We then propose a simple framework for boosting prior knowledge in streaming Bayesian learning (BPS), a variant of SVB. Our
framework emphasizes the role of the prior by encoding it inevery step, therefore the valuable prior can be efficiently
exploited in the entire learning process.
 We conduct experiments in four scenarios: streaming learningwith synthetic data, streaming learning for unsupervised sentiment analysis [17], streaming learning for latent Dirichlet allocation (LDA) [18], and streaming classification [23,24].
Comparing with the framework that does not keep prior, BPS
gives a significant improvement in performance. We further
find that the improvement of BPS over SVB is often with a large
margin for short text. This suggests that an appropriate
exploitation of prior human knowledge in streaming environments will provide significant benefits, especially for short text
and sparse data.

2. Background2.1. Variational inference
Variational inference is an approximate technique to solve
Bayesian inference problem [35]. Formally, datum x is assumed
to be generated from a Bayesian model with the prior g, the model
parameter b and the hidden variable z. The task is to find the valueof b and z to maximize the log likelihood function of the observed
data x:log pðxjgÞ
In the variational inference, instead of searching for the whole
space of b and z, we seek a variational distribution qðb; zj; cÞ,which approximates pðb; zÞ. We have:1.1. Further related work

log pðxjgÞ ¼ log

Many researches have gained significant evidences in improving Bayesian models by incorporating prior knowledge. Diamondand Kaul [25] showed the effect of using prior from megatrials in
Bayesian analysis for clinical data. Alfaro and Holder [26] usedprior in phylogenetics to learn a Bayesian model of biological data.

RRb;zpðx;b;zjgÞ:qðb;zj;cÞ
dbdz
qðb;zj;cÞ

¼ Eq ½logðpðx; b; zjgÞ  Eq ½logðqðb; zj; cÞqðb; zj; cÞþ Eq ½log
pðb; zjg; xÞ
144

ð1Þ
ð2Þ

Neurocomputing 424 (2021) 143–159

D.A. Nguyen V.L. Ngo, K.A. Nguyen et al.
Table 1
Some basic notations.
xCib; z; U

Table 2
Some distributions in the exponential family.

jjXjj
; c
n
~
ni

a data pointa minibatch – a collection of data points
model’s paremetersprior‘1 norm of X
variational parameter symbols
natural parameter of an exponential familylearned information of a minibatch i

i.i.d
s
r

independent and identically distributed
scale ratio
boosting rateg

Distribution

Parameters

Sufficient statistic
TðxÞBernoullippln 1px

PoissonNormal

k

l; r2

ln k
"
#

x


Gamma

a; b

Multinomial

p1 ; . . . ; pk
Pk
1 pi ¼ 1

Dirichlet

a1 ; . . . ; akAssuming that both b and z are continuous, otherwise integration is replaced by summation.
Note that the element in (2) is the Kullback Leibler (KL) divergence [36] between the approximate variational distribution
qðb; zj; cÞ and the true distribution pðb; zjg; xÞ which hasKLðqjjpÞ P 0. Now denoting the element in (1) by Lð; c; g; xÞ, we
get:

log pðxjgÞ ¼ Lð; c; g; xÞ þ KLðqjjpÞNatural parameter

g

l
r2


2

1
2r2

a1



b

3
ln p1
4... 5
ln pk
2
3

a1

4...5

ak

x
x2






ln x
x
2
3
x1
4...5
x
2 k 3
ln x1
4... 5ln xk

ð3Þ

so that:log pðxjgÞ P Lð; c; g; xÞFig. 2. Adding prior of exponential family.

Lð; c; g; xÞ is a lower bound of the likelihood function. Maximizing the lower bound L is equal to minimizing the KL distance
between the variational distribution q and the posterior p. The
form of q is selected in order to make L to be tractable or easier
to optimize than the original likelihood function. The solution q
is an approximation for the posterior p; hence:composed from K topics of V words, b ¼ ðb1 ; . . . ; bK Þ, each of which
is drawn from a V-dimensional Dirichlet distribution:bk  DirichletðgÞ. Each document d is a mixture of those topics
and is presumed to be created by the following generative process:pðb; zjg; xÞ  qðb; zj; cÞ

1. Draw topic mixture hja  DirichletðaÞ

2.2. Additive property of the prior of exponential family

2. For the i word in document d:
 Draw topic index zi jh  MultinomialðhÞ
 Draw word wi jzi ; b  Multinomialðbzi ÞIn probability and statistics, exponential family is a popularclass of distributions that subsumes many common distributions
including Bernoulli, Poisson, normal, gamma, multinomial andDirichlet distributions (Table 2). Their density function is of theform:

The learning problem of LDA is to estimate the posterior
pðh; z; bjC; a; gÞ of the latent variables. Nonetheless, this problem
is intractable. Applying variational inference, we approximatethpðh; z; bjC; a; gÞ  qðh; z; bjc; /; kÞ ¼ qðhjcÞqðzj/ÞqðbjkÞ

f X ðxjgÞ ¼ hðxÞ expððg  TðxÞ  AðgÞÞ

where c; /; k are the parameters of the following variational
distributions:where g is the natural parameter, TðxÞ is the sufficient statistics,hðxÞ is a known function, and AðgÞ is a normalizing element.
Consider an observed datum x generated from an exponential
distribution with prior which is encoded in the natural parameterg, and we know one more information that x is also contributed^ in the same family of distribuindependently by another prior gtion, see Fig. 2. We have:~ Þ / pðxjgÞpðxjg
^Þpðxjg; g^^ Þ  TðxÞÞ
/ hðxÞhðxÞ expððg þ gqðhÞ ¼ DirichletðhjcÞ
qðbÞ ¼ MultinomialðbjkÞ
qðzÞ ¼ Multinomialðzj/Þ
P
and k /dkv ¼ 1. According to [18], we have the lower bound of thelog likelihood of C:

ð4Þ
ð5Þ

^ Þ as its density function. The new prior for x has
which has f X ðxjg þ g
the natural parameter that is the sum of the two natural parame^ , this is the additive property of exponential family.
ters: g and g

L ¼
¼

X
d
X



Eq logðwd ; hd ; z; bÞ  Eq log qðc; /; kÞEq ½log pðwd jzd ; bÞ þ Eq ½log pðzd jhd ÞdEq ½log qðzd Þ þ Eq ½log pðhd jaÞ  Eq ½log qðhd Þ

2.3. Latent Dirichlet allocation

þEq ½log pðbjgÞ  Eq ½log qðbÞWe briefly describe latent Dirichlet allocation (LDA) [18], a wellknown model, which will be referred in several parts later.
Formally, LDA (Fig. 3) is a generative model for modeling text
data. It assumes that a corpus C containing M documents is

Taking partial derivative of L with respect to / and c and set
them to zero, we have the following inference equations for local
variables:
145

D.A. Nguyen V.L. Ngo, K.A. Nguyen et al.

Neurocomputing 424 (2021) 143–159

3.1. Streaming variational Bayes
Streaming data is considered as a sequence of minibatches
fC i gi¼1!b , where b can be infinite. Each instance is assumed to begenerated from a Bayesian model with parameter U. The learning
problem is to find fUg that maximizes the posterior:pðUjC 1 ; C 2 ; . . . ; C b Þ

ð9Þ

There is a general framework for computing the posterior in Eq.
9. The idea is to use the prior from the past data and informationfrom the current data. Fig. 4 provides a graphical representationof their idea.
Given the prior g, presuming that b  1 minibatches have beenprocessed, the posterior after b minibatches can be calculatedusing Bayes rule:Fig. 3. Graphical representation for LDA.

Algorithm 1 Inference of local variables in LDA
Input: Document d, global variable k
Output: Local variables cd , /dRandomly initialize cd ; /d .
repeatCompute /d as in Eq. 6
Compute cd as in Eq. 7
until convergence:pðUjC 1 ; C 2 ; . . . ; C b1 ; C b ; gÞð10Þ

/ pðC b jU; gÞpðUjC 1 ; C 2 ; . . . ; C b1 ; gÞpðUjC b ; gÞ/pðUjC 1 ; C 2 ; . . . ; C b1 ; gÞpðUjgÞ

ð11Þ
ð12Þ

Which means that the posterior after b minibatches is proportional to the product of the likelihood of current data pðC b jU; gÞand its prior pðUjC 1 ; C 2 ; . . . ; C b1 ; gÞ. Note that the prior element isalso the posterior of b  1 minibatches, so the posterior can becomputed sequentially. Unfortunately, the posterior in Eq. 11 isoften intractable to precisely compute. This problem leads to theneed for an approximation method. Using the variational inference,
we approximate:

Algorithm 2 Learning LDA using variational inferencepðUjCÞ  qðUjnÞ / expðn  TðUÞÞ

Input: Prior g, hyper-parameter a, data C
Output: kInitialize: k0grepeatInference:for each document d in C do
Infer /d ; cd by Algorithm 1
end forUpdate:P P
kkv ¼ g þ d v ndv /dkv
b /kuntil convergence

ð13Þ

and:

pðUjC 1 ; C 2 ; . . . ; C b1 ; gÞ  qðUjnb1 ÞpðUjC b ; gÞ  qðUj^nb Þ

ð14Þ
ð15Þ

Assume that the initial prior pðUjgÞ ¼ qðUjn0 Þ. Here the prior g
is transformed to a natural parameter of an exponential family:
n0g, so n0 is also called as the initial prior knowledge. Now combining these approximates with Eq. 12, we have:qðUjnb Þ qðUj^nb ÞqðUjnb1 ÞqðUjn0 Þ

ð16Þ

Taking the log function for both sides and use (13), we obtain/dkv / expfEq ½log hdk  þ Eq ½log bkv gX
cdk ¼ a þ ndv /dkvv

nb ¼ ð^nb  n0 Þ þ nb1

ð6Þ
ð7Þ

Denote:^nb  n0 ¼ n~b

Similarity, by taking the partial derivative of L with to k and set
it to zero, the update equation for k is:XX
kkv ¼ g þ
ndv /dkv
d

v

ð17Þ

ð18Þ

We have:nb ¼ n~b þ nb1
¼ n~b þ . . . þ n~1 þ n0

ð8Þ

where ndv is the frequency of word v (v 2 f1; . . . ; Vg) appearing in
document d.
We sum up the inference procedure using variational inference
for LDA in Algorithm 1 and the learning algorithm for LDA in Algorithm 2.

ð19Þ
ð20Þ

Eq. 19 is the streaming update function of the SVB framework[1]. It shows that the current model’s parameter is equal to thesum of the past parameter nb1 with the learned information n~bfrom current data, and hence SVB provides us a principled way tomodel streaming data without revisiting past data.

3. Revisiting streaming variational Bayes
3.2. The problem of vanishing prior
In this section, we briefly review the streaming variational
Bayes framework (SVB) [1] for learning a Bayesian model from adata stream, then discuss the problem of vanishing prior in thisframework.

A drawback in the SVB framework is the vanishing prior problem. In other words, the information from the prior is probably
vanished through learning process. This subsection will show this146

Neurocomputing 424 (2021) 143–159

D.A. Nguyen V.L. Ngo, K.A. Nguyen et al.

Lemma 3.2 (Other models). Assuming that the learned information~h gb are i.i.d samples from a probability distribution with meanfn
h¼1
n – 0, we have the following property for SVB with probability 1:


jjnb  n0 jj ¼ jjn1 þ . . . þ nb jj ! þ1 as b ! þ1;


jjn 1 þ...þn b jj
! þ1 as b ! þ1;
jjn0 jj

Fig. 4. Graphical representation of SVB for learning from streaming data.

suggesting that as b increases, the prior n0 quickly becomes vanishing incomparison with the learned information n~1 þ . . . þ n~b from data.

phenomenon for LDA model in particular, and for any model
learned by SVB.
^ ¼ k, we have the following
Replacing Eq. 18 with n0 ¼ g and n

Proof. Using the law of large numbers, we have:for the bth minibatch:~nb ¼ ^nb  n0 ¼
kv
kv
kv

X

Pr

/dkv ndv :

ð21Þ

d2C b

jj~n1 þ . . . þ ~nb jj¼ jjnjj
Pr limb!þ1bnb
sent the learned information from minibatch data b; ~n1 þ . . . þ ~represents the total information learned from the data, and n0 represents the information from the prior.
We will analyze the ratio between the norms of information
learned from the data and the information from the prior.



h

ð25Þ

This lemma shows that, in general using SVB in streaming learning will lead to the problem of losing the prior information.
Although the assumption of Lemma 3.2 is not always met, theresult provides a significant message for practice of streaming
learning, especially when we have valuable prior knowledge about
the domain/task of interest.
Considering the circumstances that we have valuable priorhuman knowledge that describes properties of data, losing theinformation of these priors in the learning process is highly wasteful. Bayesian methods often assume the data to be generated froma probabilistic process. If no knowledge is present, those methods
can learn information efficiently only from data. However, notethat most models are mis-specified, and as a consequence, might
not model the data exactly. From this reason, combining priorhuman knowledge into a Bayesian model is an essential method
to improve the quality of Bayesian models.
To overcome the problem of vanishing prior in SVB, we proposea framework for Bayesian learning to maintain the valuable information from the initial prior.

ð26Þ

4. Boosting prior knowledge in streaming Bayesian learningð22Þ

jjn 1 þ...þn b jj
! þ1 as b ! þ1;
jjn0 jj

suggesting that as b increases, the information from the prior n0 quickly
becomes vanishing in comparison with the learned informationn~1 þ . . . þ n~b from data.
Proof. It is clear that jjn0 jj is a constant. We now provePnb jj ! þ1asb ! þ1.
Becauseandjj~
n1 þ . . . þ ~k /dkv ¼ 1
n~b kv P 0, from Eq. 21 we have the following for any b P 1:

jjn~b jj ¼

XX
k

v

n~bkv ¼¼

XXX
/dkv ndv
d2C

k

d2C

v

XX

X
¼
nd

ð23Þ

v

X
ndv
/dkv

¼ 1;

!
jj~n1 þ . . . þ ~nb jj
Pr lim¼ þ1 ¼ 1:b!þ1
jjn0 jj!

and because jjn0 jj is a constant, we have:

Lemma 3.1 (LDA). The SVB update of the LDA model has thefollowing property:!
~n1 þ . . . þ ~nb¼ n ¼ 1;
b!þ1
b
lim

suggesting thatAs a result, the prior n0 in SVB is used only one time in the initial~b represtage as in Eq. 20. It is clear that the learned parameters njjnb  n0 jj ¼ jjn1 þ . . . þ nb jj ! þ1 as b ! þ1;

ð27Þ

ð24Þ

k

d2C

P 1:
So

In this section, we first present the BPS framework for boosting
the valuable prior knowledge in SVB. We then propose to choose
the linear boosting function and analyze some advantages of sucha choice. Finally, we discuss how to balance between boosted priorand the knowledge learned from each minibatch of data.

jj~n1 þ . . . þ ~nb jj P b;

which means jjn1 þ . . . þ nb jj ! þ1 as b ! þ1.

h

The result in Lemma 3.1 can be interpreted as following: In SVB
with the initial prior only exists in the first stage, through thestream learning process, the new knowledge is added into themodel’s parameter. When data is big enough, almost all information in the model’s parameter comes from the data and the roleof the initial prior will vanish quickly. This problem can be easilly
seen in LDA.Next, we discuss the vanishing prior phenomenon of
SVB in broader contexts. For more complex models, such a phenomenon is not easily observed. We have to employ the wellknown law of large numbers under some assumptions. Formally,
we have the following.

4.1. The BPS framework^
The idea of BPS is to directly add one more prior information g^bto each minibatch. Fig. 5 depicts this idea clearly. The new prior gof the minibatch b is a boosting function of the initial prior g sothat it reflects how the prior knowledge impacts to each minibatchof the streaming learning process.

g^ b ¼ ^f b ðgÞ

147

ð28Þ

D.A. Nguyen V.L. Ngo, K.A. Nguyen et al.

Neurocomputing 424 (2021) 143–159

gg0 ¼@ðexpðg  TðUÞÞ@ðg  TðUÞÞ
¼ expðg  TðUÞÞ
@ðUÞ
@ðUÞ

and

g 0rg ¼

Obviously, g 0g ¼ 0 and g 0rg ¼ 0 have the same set of solutions
which means both pðUjgÞ and pðUjrgÞ have the same feature points
Uf .
In other aspect, we interpret the value of g0 near a feature point
as the changing of the concentrated density around the featurepoint. The big value of g0 means that there is a high concentrated
density level or a high important level.
Assuming g  TðUÞ > 0, it is clear that if r < 1, then jg0rg j < jg 0g j,
we decrease the important level of feature points. Otherwise, if
r > 1, then jg0rg j > jg0g j, we increase the important level of featurepoints. In another words, by changing the value of boosting rate in
the linear boosting function, we can change the important level offeature points of the prior distribution.
To illustrate this property, we consider an example with Dirichlet distribution on the 2-simplex x1 ; x2 ; x3 :Fig. 5. Graphical representation for BPS.

An additional prior in the form of an exponential family^ðUjg^b Þ / expðg^b  TðUÞÞ is used in each minibatch. To emphasizep^ b , we will optimize the following
the role of the new prior g
probability:^ b ÞpðC b jU; gÞpðUjC 1 ; C 2 ; . . . ; C b1 ; gÞpðUjg
^ b ÞpðUjC b ; gÞpðUjg¼pðUjC 1 ; C 2 ; . . . ; C b1 ; gÞpðUjgÞ

ð29Þ
ð30Þ

With ^
nb0g^ b , using the additive property of prior for exponential family and the variational method, we have an approximation:^ b ÞpðUjC b ; gÞ / pðUjC b ; g^ b þ gÞpðUjg qðUjn^b Þpðx1 ; x2 ; x3 Þ
¼ x1 x12 x3 expð

ð31Þ
ð32Þ

ð^nb  n0 Þ þ nb1

ð33Þ

Supposing that ^nb can be decomposed into the sum of the
~
and
the
prior
n0 þ ^
nb ,i.e.,learnedinformationnb^
nb þ ðn0 þ ^nb0 Þ, then Eq. 33 becomes:
nb ¼ ~

nb ¼ ~nb þ ^nb0 þ nb1

3
X

i¼1

i¼1

gi lnðxi Þ 

3X
ln Cðgi Þ  ln Cð gi ÞÞi¼1

4.2.1. Relation to power prior
In BPS, we observe that expðrg  TðUÞÞ ¼ ½expðg  TðUÞÞr . This isequivalent to the use of the variational distribution of the following
form

0

ð34Þ

^ Þ / qðUjgÞr
qðUjgThe difference between the streaming update function in Eq. 34
of BPS and the one of SVB in Eq. 19 is that there is an additional ele^b . This property means that BPS boosts
ment of the boosting prior nð36Þ

In fact, linear boosting means that we use a power scale of theinitial prior. This concept is related with the power prior method in[32,33,3], but BPS has some significant differences.
In power prior, presume the model’s parameter U has the initial
uninformative prior pðUjgÞ, a past data C 0 and a power factor q.
The power prior is defined by:0

the knowledge from the initial prior through streaming learning^ b ¼ 0, the BPS will become SVB.
process. If we set g
4.2. Linear boosting function
In BPS, we introduce a boosting function in Eq. 28 that describes
how the original prior impacts on the learned model. Now we propose a simple form of the boosting function in term of linear relationship with a boosting rate rb 2 R:g^ b ¼ ^f b ðgÞ ¼ rb g

3
X

In this example, the natural parameter g ¼ ½4; 5; 6, and theboosting rates are tested with: 0.5, 1 and 1.5 respectively. The
results are shown in Fig. 6. Intuitively, the figure shows that thebigger boosting rate, the more concentrated level of the density.

Replacing Eqs. (32)–(30), we have the new update function:nb@ðexpðr g  TðUÞÞ
r@ðg  TðUÞÞ
¼ expðr g  TðUÞÞ
@ðUÞ
@ðUÞ

pðUjC 0 ; gÞ / LikelihoodðUjC 0 Þq :PriorðUjgÞ

ð37Þ

¼ pðC 0 jUÞq :pðUjgÞ:ð38Þ

Receiving new data C leads to the posterior:pðUjC; C 0 ; gÞ / pðCjUÞ:pðUjC 0 ; gÞ

ð35Þ

¼ pðCjUÞ:pðC 0 jUÞq :pðUjgÞ:Assuming g is the natural parameter in the exponential form ofprior knowledge pðUjgÞ / expðg  TðUÞÞ, we will show that with thelinear function in Eq. 35, the new prior pðUjr gÞ will keep somemain features of the original prior distribution with a level ofimportance.
Definition: The feature points Uf of a probability distributionpðUjgÞ are defined by its local extrema.
The feature points Uf can be used to describe the shape of theprior distribution, containing stationary points and boundary
points. The local maximal points are positions that have high density, and the local minimal points have low density.
Taking derivative of the density functions of pðUjgÞ and pðUjrgÞ
over U, we have:ð39Þ

The power prior depends on the past data and the power element is the likelihood of the past data C 0 , while BPS straightforwardly powers the initial prior. On the other hand, the power
prior method modifies the prior for the new data and replacescompletely by the new prior specified in Eq. 38. By this way, thepower prior provides a way to balance the old knowledge (learned
from past data) and the new knowledge (just learned from C of thecurrent step). The balancing constant q needs being chosen manually. Masegosa et al. [3] make a further progress by considering thebalancing constant q to be a random variable which follows a priordistribution, allowing q to be adaptive with the data stream. In

148

Neurocomputing 424 (2021) 143–159

D.A. Nguyen V.L. Ngo, K.A. Nguyen et al.

Proof. Using the law of large numbers, we have:!
~n1 þ . . . þ ~nb
Pr lim¼ n ¼ 1;b!þ1b
and because jjn0 jj is a constant:jj~n1 þ . . . þ ~nb jj
jjnjjPr lim¼
b!þ1
bcjjn0 jj
cjjn0 jj!
¼ 1:

~ b ¼ r b n0 and r b P c for any b, we have:
Note that g
jj~n1 þ . . . þ ~nb jj
jjnjj
Pr lim6~1 þ . . . þ g~ b jj cjjn0 jj
b!þ1 jjn0 þ g

!

¼ 1:

This lemma shows that, in BPS, the information from external
prior is not overwhelmed by the information learned from data.
Hence, we can remain the valuable knowledge.

4.4. Balancing between learned information and boosting prior
BPS aims to inject the information from the initial prior into thenew data. In Eq. 34, the learned information from the new data n~b is^b . Because n~b depends on the amount of
independent of n0 and n0data, overusing the boosting prior may leads to underfitting
because it dominates the learned information. To tackle this problem, we propose an information scale ratio between jjn~b jj and jj^
nb jj:
0jj^nb0 jj ¼ sjjn~b jj

ð41Þ

where s is a scale parameter. The purpose of this balancing is to let
the information from the prior knowledge account for a certain
ratio in the learned information of the new data.
Suppose we use the linear boosting function in Eq. 35. Note thatn0g and ^nb0 g^ b , combining with Eq. 41 leads to the boostingrate:

rb ¼ sFig. 6. Dirichlet distributions on the 2-simplex with different boosting rates ofnatural parameter g ¼ ½4; 5; 6.

jjn~b jj
jjn0 jj

ð42Þ

and the boosting prior:~^nb ¼ s jjnb jj :n00
jjn0 jj

contrast, the viewpoint of BPS is to inject human/external knowledge into a model at each time step.

In practice, at first we compute the learned information n~b , thenwe follow Eq. 43 to get the boosting prior. Finally, we update theparameter using Eq. 34. In BPS; s is a hyperparameter, we canchoose a good value for s by grid search in the range ½0; 1.
Note that with scale factor s ¼ 0; BPS turns out to be SVB without using boosting prior.

4.3. Keeping prior in BPS
We will show that in BPS with linear boosting functions, information from prior is not vanished. In BPS, the information learned
from the data is: ~n1 þ . . . þ ~nb and the total information from prior~1 þ . . . þ g~ b . By evaluating ratio of the norms of these two
is n0 þ gkinds of information, we have the following lemma:

4.5. Discussion about BPS

Lemma 4.1 (BPS). Assume that the learned information f~
nh gh¼1 are i.
i.d samples from a probability distribution with mean n – 0, and BPS~ b ¼ rb n0 , where coefficients r b areuses the linear boosting function glower bounded by some constant c > 0. We have the following
property, with probability 1,b

jjn1 þ . . . þ nb jj

jjn0 þ g1 þ . . . þ gb jj

BPS is a generalization of SVB with the ability to appropriately
use prior knowledge. While a self-contained model to cover thedata with high performance is not available and we often spenda high cost on deriving a good model, BPS is a considerable solution. In cases of having a good prior knowledge, BPS is a straightforward way with low cost to enhance a Bayesian model and reducesome bad effects from extreme sparsity and noise. One can alsointerpret that BPS uses external knowledge to regularize a model
when dealing with ill-posed problems, e.g., sparsity and noise.
We observe that external/human knowledge is available in various
forms and easily accessible, including pre-trained machine learning models [22,37,38], ontology, Wikipedia, Zipf’s law, Wordnet,
etc. Furthermore, we believe that the idea of BPS can be easily6

jj n jj
as b ! þ1;
cjjn0 jj

ð43Þ

ð40Þ

suggesting that as b increases, the prior information~1 þ . . . þ g~ b is not vanished in comparison with the learned
n0 þ ginformation n~1 þ . . . þ n~b from data.
149

D.A. Nguyen V.L. Ngo, K.A. Nguyen et al.

Neurocomputing 424 (2021) 143–159

employed in other streaming frameworks [2,19,3,5]. Those observations suggest the solution from BPS is significant.
Nonetheless, BPS requires a good prior knowledge to improve amodel, otherwise, it may worsen the model. This consequence is
explicitly shown in a case study of Section 5. In some situations,obtaining prior knowledge with good quality may be hard, andtherefore BPS has some limitations.

5. Case studies
In this section, at first we conduct a case study with synthetic
streaming data and then we build 3 other case studies with 3 models and different type of prior knowledge to evaluate the performance of BPS framework in comparison with SVB. The streaming
data is simulated by dividing a dataset into continuous collections.
For selecting models’ hyperparameters, we do a grid search aroundtheir recommended settings in its original papers. For BPS, we first
do a grid search to find a good range of scale factor s then evaluate
how BPS responses to the change of s. We also do experiments on
the sensitivity of BPS over different values of the batchsize parameter. Each experiment is run five times with random initializationof variables. The final result is the mean value of all runs. For simplicity, we only plot standard deviation of experiments in several
cases. In all cases, higher values are better.
5.1. Case study 1: streaming with synthetic data
In this case study, we examine the performance of BPS in comparison with SVB in a synthetic streaming data, and investigate theimpacts of the quality of prior knowledge and noise.
Consider a model where the binary data are all generated from a
Bernoulli distribution: x  BernoulliðhÞ. We fix h ¼ 0:2 and generatea sequence of 300 data points.
Suppose that we use the following misspecified model to work

Fig. 7. Impact of prior knowledge quality. ’True’ is the correct value (0.2) of model
parameter h in the BB model that generates the training data. Some guesses {0.1,0.2, 0.3, 0.4, 0.5} about the true h are used as priors for SVB and BPS. E½h shows how
well a method recovers h as learning from more data.

cwith those data: h ¼ Betaðgc ; gd Þ; x  BernoulliðhÞ. Note E½h ¼ gcgþgd ,where gc and gd are hyperparameters.
This model is denoted as BB. The learning involves the following
posterior: pðhjgc ; gd ; CÞ. Denote nb and kb be the number of data
points and the number of 1’s in the data C b of minibatch b, respectively. Using variational inference with the variational distribution

Result: The result is illustrated in Fig. 7a for SVB and in Fig. 7bfor BPS. It shows that SVB has less impact of prior knowledge while
BPS has a strong dependence with the quality of prior knowledge. A
good prior knowledge will significantly enhance the quality ofmodel. In contrast, a poor quality of prior knowledge misleads
the model, and even makes it poorer than SVB. Therefore, the quality of prior knowledge has a real impact to BPS, and BPS only keeps
its good performance in case of having a good prior.

qðhÞ  Betaðkc ; kd Þ, a lower bound of the log likelihood of the datais:

L ¼ Eq log pðh; Cjgc ; gd Þ  Eq log q
Similar to the learning method of LDA, by taking the partial

5.1.2. Learning from noisy data
Suppose that some random noises change the results of Bernoulli trials. In more details, the noisy generative process for eachdata point is:derivative of L with respect to kc and kd and setting them to zero,
we have Algorithm 3 and Algorithm 4 for learning BB by SVB andBPS respectively.

 Sample data point x  BernoulliðhÞ
 Change the value of x with a noise BernoulliðdÞ.2

5.1.1. Impact of prior knowledge quality
We evaluate the impact of prior knowledge quality into the performance of BPS by setting different levels of prior knowledge. The

With the same settings as sub-Section 5.1.1, but only using thegood prior with the expectation E½h ¼ 0:2 and the noisy probability
d 2{0.01, 0.05, 0.1, 0.2, 0,3}, we have the results in Fig. 8a for SVB
and in Fig. 8b for BPS.
It is clear that in noisy data, BPS with a good prior knowledge
will drive the model into its true value. Meanwhile, SVB is
impacted by wrong information from noise leading to a weak performance. In other words, BPS has ability to help the model copewith noisy information when providing a good prior knowledge.

cknowledge to be used in BPS is encoded in ðgc ; gd Þ. Note that gcgþgd
being closer to h ¼ 0:2 means that the prior quality is better.
Settings: The batchsize is fixed to 20, and the scale factor of BPS
is s ¼ 0:3.
Evaluation: We compare the expectation of h in the learned
model with its true value. The closer h is to the true value, the better the model is.
Prior in use: We investigate different priors ðgc ; gd Þ so thatc^h ¼ cg d 2 f0:5; 0:4; 0:3; 0:2; 0:1g. The closer ^h to the truth h ¼ 0:2,g þg

the better the quality of prior knowledge.

2150Note that the contaminated x turns out to be a trial from Bernoulliðh þ d  2hdÞ.

Neurocomputing 424 (2021) 143–159

D.A. Nguyen V.L. Ngo, K.A. Nguyen et al.

5.2. Case study 2: streaming sentiment analysis
Sentiment analysis is an interesting area in text mining, withthe aim to extract opinions from text data. One of the basic tasks
in sentiment analysis is sentiment classification, in which opinions
often need to be classified into positive, negative or neutral. However, a document can contains many aspects with different sentiments. Yo and Oh [17] proposed the Aspect and SentimentUnification model (ASUM) for modeling the sentiments about
aspects in an unsupervised manner. The graphical model of ASUM
is presented in Fig. 9.
ASUM assumes that a corpus is talking about E sentiments and T
aspects, represented by many Dirichlet distributions bez  Dirðge Þ,
where e 2 f1; . . . ; Eg is the index of a sentiment and z 2 f1; . . . ; Tg
is the index of an aspect. In each document, containing M sentences, the distribution of sentiments is pd  DirðcÞ; and for eachsentiment e, the aspect distribution is defined by hde  DirðaÞ. We
assume the mth sentence of length N is generated as follow: firstchoose the sentiment by a multinomial distribution em  Mulðpd Þ,then generate the corresponding aspect zm  Mulðhdem Þ, generateeach word w  Mulðbem ;zm Þ.
Learning algorithm: Given data C, the full posterior of interestis:pðb; p; h; e; zjC; c; a; gÞð44Þ

The inference procedure infers the local variables p; h; e; z. We
use the method proposed by [39]. The Dirichlet parameters c ¼ 1
and a ¼ 1 are fixed so that the posterior in Eq. 44 is a convex function with respect to p and h and can be efficiently inferred byFrank-Wolfe algorithm [40]. e and z can be directly approximatedthrough pðejp; h; w; b and pðzjp; h; w; bÞ. The details of this procedure is presented in Algorithm 12 of the Appendix.
To update the global variable b, let qðbjkÞ be the variational distribution of pðbjgÞ. Using variational inference, we derive the two
streaming learning algorithms for ASUM, SVB in Algorithm 5 and
BPS in Algorithm 6.
Prior knowledge in use: We use the same way as in [17,39].
We use a list of positive (e.g., ‘‘good”, ‘‘excellent”) and negative
words (e.g., ‘‘bad”, ‘‘poor”) as prior knowledge, often called seedwords.3 be contains information about the distribution of words
over each sentiment. So for the seed words, the priors are set withhigher value than other words. By this way, we form a humanprior of sentiment over words: ge . Therefore, the prior encodedin BPS is

Fig. 8. Impact of noise. ‘True’ is the correct value of model parameter h in the BB
model that generates the training data. {0.01, 0.05, 0,1, 0.2, 0.3} show the level ofnoise that contaminates the training data. E½h shows how well a method recovers h
as learning from more noisy data.

Algorithm 3 SVB-BB
Input: Prior gc ; gd , sequence of mini-batch C 1 ; C 2 ; . . .
Output: a sequence kcb ; kdb
Initialize: kc0
gc , kd0 gd
for each mini-batch C b in C 1 ; C 2 ; . . . do
~
kb
kc
b

~
kcb
kcb

nb  kb~dkc þ k~^f ðgÞ ¼ s jjkb jj g:
b
jjgjj

b1
b
kdb1 þ ~kdb

kdb
end forWe set gej ¼ 0:01 for any sentiment seed word j and otherwise
gei ¼ 0:0001 for any other word i. ge is used for both SVB at the firstlearning step and BPS.
Evaluation metric: The accuracy of sentiment classification isused to evaluate the quality.
Experimental setups: We use 4 datasets (Electronics, Yelp,
Kitchen & Housewares, and Music)4 with some statistics shown in
Table 3.
To investigate the effect of scale ratio s, we fix batch-size to1000 for the three first datasets, and to 2000 for Music as it’s sizeis large. We test s 2 f0:2; 0:4; 0:6; 1:0g. To study the effect of batchsize, we fix the scale ratio to s ¼ 0:4 and change batch-size in
f500; 1500; 2000g for the three small corpora, and in
f4000; 6000; 8000g for Music. We have E ¼ 2 to represent

Algorithm 4 BPS-BB
Input: Prior gc ; gd , sequence of mini-batch C 1 ; C 2 ; . . .
Output: a sequence kcb ; kdb
Initialize: kc0
gc , kd0 gd
for each mini-batch C b in C 1 ; C 2 ; . . . do~f ðgc Þ þ kb
kc
b

~
kcb
kcb

b

f b ðgd Þ þ nb  kb~dkc þ kb1

b

kdb
kdb1 þ ~kdb
end for3
4

151

The full list can be found in [39].
http://www.cs.jhu.edu/~mdredze/datasets/sentiment/

D.A. Nguyen V.L. Ngo, K.A. Nguyen et al.

Neurocomputing 424 (2021) 143–159

Results:
The effect of prior: Fig. 10 shows the results of BPS with different
scale ratios, in comparison with SVB. Higher value of the scale ratiomeans we are using more prior knowledge.
It is clear that the results of BPS are better than those of SVB
even with different values for scale ratio. In addition, there are risesin the accuracy of BPS when the scale ratio increases. It means thatthe prior is meaningful, so that emphasizing it will enhance thequality of the learning process. Moreover, Fig. 10b and Fig. 10d give
illustrations that ignoring the external prior in SVB may lead to aconsiderable decrease in accuracy, while by maintaining the prior,
BPS improves the accuracy of the model. Those subfigures also suggest that by boosting the good prior, BPS can perform better by alarge magin than SVB.
Fig. 11 summarizes the results when changing the batch-size.
We observe that BPS works well with different settings of batchsize. In all 4 datasets, the accuracy of BPS with different batchsizes is higher than that of SVB. The performance of BPS in Yelp
and Music is particularly good, and significantly better than thatof SVB. The improvement with a large margin in Music suggeststhat the prior is really meaningful for sentiment analysis. Losing
such a good prior knowledge causes to quickly reduce the discriminative ability of SVB.

Fig. 9. The graphical representation of ASUM.

positive/negative sentiments, and fix T ¼ 50 in all experiments for
ASUM.5

Algorithm 7 SVB-LDA
Algorithm 5 SVB-ASUM

Input: Prior g, hyper-parameter a, sequence of minibatches
C1; C2; . . .
Output: kInitialize: k0gfor each minibatch C in C 1 ; C 2 ; . . . doInference:for each document d in C do
Infer local variables /; cend forUpdate:P/ n dv
k~b

Input: Prior g; a; c, sequence of mini-batch C 1 ; C 2 ; . . .
ðlÞ

ðlÞ

Output: a sequence k1 ; k2 ; . . .
gInitialize: k0
for each mini-batch C in C 1 ; C 2 ; . . . doInference:for each document d in C do
Infer local variables ðh; p; e; zÞ
end forUpdate:P
P P  kb etj ¼ d2C b m n edme z dmt djkb
kb1 þ k~bd2C

bb / kb
end for

Algorithm 8 BPS-LDA

Algorithm 6 BPS-ASUM

Input: Prior g, hyper-parameter a, sequence of minibatches
C1; C2; . . .
Output: kInitialize: k0gfor each minibatch C in C 1 ; C 2 ; . . . doInference:for each document d in C do
Infer local variables /; cend forUpdate:P/ n dv
k~b

Input: Prior g; a; c, sequence of mini-batch C 1 ; C 2 ; . . .
ðlÞ

ðlÞ

Output: a sequence k1 ; k2 ; . . .
gInitialize: k0
for each mini-batch C in C 1 ; C 2 ; . . . doInference:for each document d in C do
Infer local variables ðh; p; e; zÞ
end forUpdate:P
P P  kb etj ¼ d2C b m n edme z dmt djkkþ k~ þ ^f ðgÞ
b

b1

b

dv k

kb
kb1 þ k~b
bb / kb
end for

d2C

dv k

kb
kb1 þ k~b þ ^f b ðgÞ
bb / kb
end forbbb / kb
end for5
We found the same conclusion as investigating with different values of T. Hencefor ease of discussion, we just report the results with T ¼ 50.

152

Neurocomputing 424 (2021) 143–159

D.A. Nguyen V.L. Ngo, K.A. Nguyen et al.
Table 3
Datasets for evaluating ASUM. ld denotes the average number of words per review. ls denotes the average number of words per sentence.
Dataset

Reviews

Positive reviews

Negative reviews

ldlsElectronics
Yelp restaurant
Kitchen & Housewares
Music

23,00920,70819,856
136,000

17,961
17,457
15,737110,160

5,048
3,251
4,119
25,840

38.6
59.7
31.9
57.2

6.1
5.7
5.8
7.1

greater s and batch-size for short text corpora, because in thosecases more information/knowledge should be needed.
The effect of prior: The results in this case study have a slightly
different pattern with that in case study 2, and can be divided into
two groups: long text (New York Times), and short text (Yahoo andTwitter). In general, a higher weight of the prior leads to worse performance of BPS in long text, however gives significant improvement for short text (Fig. 12).
In the experiments for testing boosting scale with the long textgroup, the smallest value of scale ratio for BPS in the experimentss ¼ 0:02 gives the highest quality of the model and better than in
SVB. However, when increasing the boosting scale, the predictive
capacity of LDA using BPS may be lower than using SVB. In contrast, the performance of BPS for the short text increases asincreasing the boosting scale and gets the highest quality withs ¼ 0:2. The better performance of BPS over SVB is very clear
and often with a large margin. The same behavior of BPS isreported in Fig. 13 when changing the batch-size and fixing thescale ratio with a suitable value (0.02 for short text, and 0.08
for long text).
We can explain such a behavior of BPS as follows. The short text
often contains little information, and poses severe challenges for
SVB. Further even a large number of short text could not overcome
those challenges [11]. Therefore, exploiting external knowledge
would be necessary in order to learn a good predictive model fromshort text. BPS seems to exploit considerably well the knowledge
from Zipf’s law, represented in (45), to surpass SVB with a large
margin in Yahoo and Twitter. On the other hand, the long text itself
contains much information and hence helps SVB perform well.
Overusing prior knowledge with a unsuitable scale factor will leadto overwhelm the information from data. As a result, the role ofprior is not very significant in this case.

5.3. Case study 3: streaming learning for LDA
In this case study, we evaluate BPS’s performance with the LDA
model presented in Section 3.2.
Streaming learning algorithm: We present more detailed versions of the learning algorithms for LDA than in Section 3.2. The
posterior of the latent variables given a corpus C is: pðb; h; zjC; a; gÞ.
We inherit the learning algorithm for LDA in Section 2.3 with
the inference procedure for the local variables as in Algorithm 1.
After applying BPS and SVB framework, we have two streaming
learning algorithms for LDA in Algorithm 7 and 8. Note that Algorithm 7 is the same as the SSU algorithm in [1]. The only difference
between the two algorithms is ^f ðgÞ.
b

Prior knowledge in use: Relating to the distribution of word innatural language, Zipf’s law [27,28] gives us an interesting property that the frequencies of words in a specific language followed
a power-law distribution of the form: pðwÞ / rlw , in which pðwÞ isthe proportion of word w in the language. rw is the rank of the wordin the descending sorted frequencies. This means that the most frequency word has rank r ¼ 1. Parameter l depends on the specificlanguage. We use Zipf’s law as the prior knowledge in BPS as
follows:

gw / rlw:

ð45Þ

Evaluation metric: We use log predictive probability [1] toevaluate the predictive capacity of the learned models. A testing
corpus C h is taken out from data. Each document of this test corpus
is split into 2 parts: dtest and dobserv ed (with a ratio of 1 : 4 in thisexperiment). Suppose that the global parameters are given, thelog predictive probability (LPP) is defined by:

X
LPP ¼

d2C h

X X

log pðdtest jC;dobser v ed Þ

X
d2C h

jdtest j

d2C h w2dtest

log

K
X

5.4. Case study 4: streaming text classification

Eq ½hdk Eq ½bkw 

k¼1

XTo solve text classification problem, multi-view topic modelðMv iewLDAÞ [23] was introduced. The idea of MviewLDA is thateach document of D documents belongs to one of J classes withprobability contributed by a multinominal distribution

jdtest jd2C hExperimental setups: We use 3 datasets for evaluation: NewYork Times from the UCI Machine learning Repository6, Yahooand Twitter from [10]. Some statistics are presented in Table 4. Notethat Yahoo and Twitter are two corpora of extremely short text,
while New York Times is long text. The number of topics is set
equally with K ¼ 100, and the hyperparameter a ¼ 0:01. The priorg is taken from Eq. 45 with a heuristic parameter l ¼ 1:07 [41] andthe ranking of word’s frequencies is downloaded from top 100,000
most frequently-used English words text,7 only the words appearedin vocabulary of dataset are used.
For testing the effect of the scale ratio, we fix the batch-size= 5000 for large corpora and 1000 for New York Times, and changes 2 f0:02; 0:05; 0:08; 0:1; 0:2g. With batch-size testing, we fix
s ¼ 0:02 and change batch-size in f500; 2000; 4000g for New YorkTimes. With Yahoo and Tweet we fix the scale ratio to s ¼ 0:08
and change the batch-size in f5000; 10000; 20000g. We choose
67Kv  MulðpÞ. Each class j contains K local topics fbðlÞ
jk gk¼1 with the
distribution over topics hðlÞ which are sampled from the DirichletðlÞ

distribution Dirðaj Þ. Besides, they assume that there exists R gloRbal topics fbðgÞ
r gr¼1 that are shared by all classes. The binary variable d decides when a word belongs to global or local topics. The
graphical representation of MviewLDA is presented in Fig. 14.
The generative process of MviewLDA is as follows: Generate topic distribution:– For local topics: bðlÞ  DirichletðgðlÞ Þ– For global topics: bðgÞ  DirichletðgðgÞ Þ
 Generate a document d of length N d :– Draw a class: v  MulðpÞ
ðlÞ– Draw local topic proportion: hðlÞv  Dirðav ÞðgÞ– Draw global topic proportion: h  DirðaðgÞ Þ– Draw Bernoulli parameter x  BetaðcÞ

http://archive.ics.uci.edu/ml/machine-learning-databases/bag-of-words/
https://gist.github.com/h3xx/1976236.
153

D.A. Nguyen V.L. Ngo, K.A. Nguyen et al.

Neurocomputing 424 (2021) 143–159– For each word w of document d:⁄ Draw a binary indicator d  BernoulliðxÞ
⁄ If d ¼ 1, word w belongs to local topic:ðlÞ Draw a local topic zv  MulðhðlÞv Þ
 Draw word w  MulðbzðlÞ Þv⁄ If d ¼ 0, word w belongs to global topic:
 Draw a global topic: zðgÞ  MulðhðgÞ Þ Draw word w  MulðbzðgÞ ÞðlÞLet each local topic being contributed by a Dirichlet prior gj .
Note that local topics within each class contain feature words of
its class. Therefore, if we set the weighted value of the featurewords larger than the others and use it as a prior, we will provide
more information into the model to differentiate classes. Such atechnique can increase the quality of learning process.
Algorithm 9 SVB-MviewLDA
Input: Prior g, sequence of minibatches C 1 ; C 2 ; . . .
Output: kInitialize k0g
for each minibatch C in C 1 ; C 2 ; . . . doInference:for each document d in C do
Infer local variables ðf; /; sÞend forUpdate:P PN dðlÞðlÞ~kI½v ¼j sdi /
wd;i;j
b

d2C

i¼1

d

d;i;j;kkb
kb1 þ k~b
bb / kb
end for

Algorithm 10 BPS-MviewLDA
Input: Prior g, sequence of minibatches C 1 ; C 2 ; . . .
Output: kInitialize k0g
for each minibatch C in C 1 ; C 2 ; . . . doInference:for each document d in C do
Infer local variables ðf; /; sÞend forUpdate:P PN dðlÞ~ðlÞkI½v ¼j sdi /
wd;i;j
b

d2C

i¼1

d

d;i;j;kkb
kb1 þ k~b þ f b ðgÞ
bb / kb
end for

Learning algorithm: Given corpus C ¼ fC i g, in case of supervised learning with known class label v, the posterior of interest is:pðx; hðlÞ ; hðgÞ ; d; z; bðlÞ ; bðgÞ jv; C; gðlÞ ; gðgÞ ; aðlÞ ; aðgÞ ; c; pÞ:ð46Þ

The learning algorithm in [23] is Gibbs sampling and designed
for batch learning. In this case study, we use variational inference(as in [42]) and modify to streaming classification learning. The
variational distribution is:qðx; hðlÞ ; hðgÞ ; d; zðlÞ ; zðgÞ ; bðlÞ ; bðgÞ Þ

Fig. 10. Streaming ASUM results with different scale ratios. The x-axis shows theamount of data received in the streaming learning process.

¼ qðxjmÞqðhðlÞ jlðlÞ ÞqðhðgÞ jlðgÞ ÞqðdjsÞqðzðlÞ j/ðlÞ ÞqðzðgÞ j/ðgÞ ÞqðbðlÞ jkðlÞ ÞqðbðgÞ jkðgÞ Þ154

Neurocomputing 424 (2021) 143–159

D.A. Nguyen V.L. Ngo, K.A. Nguyen et al.
Table 4
Datasets for evaluation of LDA. ld denotes the average number of words per
document.
DatasetVocabulary size

Training size

Testing sizeld

New York Times
Yahoo
Twitter

102,660
24,41989,474200,000
500,000
1,500,000

10,00010,00010,000

228.8
4.7
9.8

Fig. 11. Streaming ASUM results with different batch-sizes.

where /ðlÞ and /ðgÞ are multinomial parameters, m is Beta parameter,s is Bernouli parameter, /ðlÞ ; /ðgÞ ; kðlÞ ; kðgÞ are Dirichlet parameters.
We denote the indicator Ijvd ¼ 1 when document d belongs to classj and Ijvd ¼ 0 otherwise. Similarly, Ivwd;i ¼ 1 if the ithword of docu-

ment d is v. The detail of the inference procedure are shown in Algorithm 11 of the Appendix.
Because we aim to keep the information from prior gðlÞ of thethe variational inference leads to 2 streaming versions for updating

local topics, we only apply BPS to bðlÞ . Adopting SVB and BPS withkðlÞ in Algorithm 9 and Algorithm 10.

Fig. 12. Streaming LDA results with different scale ratios.

155

D.A. Nguyen V.L. Ngo, K.A. Nguyen et al.

Neurocomputing 424 (2021) 143–159
Table 5
Dataset for text classification.
DatasetNum of classes

Vocabulary size

Training size

Testing size

News20
Cade1220
1262,061
193,997

16,00027,322

3,900
4000

Fig. 13. Streaming LDA with different batch-sizes.

Fig. 14. Graphical representation for MviewLDA.
Fig. 15. Streaming MviewLDA result with different scale ratios.

Prior knowledge in use: Borrowing the idea from [30], we
extract the feature words of each class and use them as the priorknowledge. At first, we calculate TF.IDF for the words in each classthen select top 5000 words with highest TF.IDF values as seedwords. The seed words of class j are then used to initialize prior

For News20, we set the number K ¼ 10 of topics in each local
class and the number R ¼ 8 of global topics. For Cade 12, we use
K ¼ 15 and R ¼ 4 because of being larger in size than News20.
The other Dirichlet prior parameters are set equal to 0.01 as casestudy 3 on LDA. We take SVM into account as another baseline
to evaluate the quality of classification.
For testing scale ratio, we set batchsize as 1000 and change
scale factor s 2 f0:2; 0:4; 0:6; 0:8; 1:0g. For testing batchsize, we fix
scale factor s ¼ 0:4 and change batchsize in f1000; 1500; 2000g.
Results: The results are summarized in Fig. 15 and Fig. 16. The
results of BPS are comparable with those of SVB.
However, we can see that the accuracy of BPS tend to be higher
than SVB in the latter minibatches, and significantly better inNews20 dataset. Again, this pattern reflects that maintaining thegðlÞ
j by assigning a value v ¼ 0:5. The other values are set to a smallvalue  ¼ 0:01.
Evaluation metric: The classification accuracy is used in thisevaluation.
Experimental setups: We use 2 labeled datasets (Cade 12 andNews20).8 with some statistics described in Table 5.

8

http://ana.cachopo.org/datasets-for-single-label-text-categorization
156

Neurocomputing 424 (2021) 143–159

D.A. Nguyen V.L. Ngo, K.A. Nguyen et al.

Appendix A
This section presents the details of the inference of local variables
in ASUM (Algorithm 12) and MviewLDA (Algorithm 11).
Algorithm 11 Inference of local variables in ASUM
Input: Prior g; a ¼ 1; c ¼ 1, global variable b, documentd ¼ fw1 ; . . . ; wm g with M sentences
Output: Local variables: h; p; ~ed ; ~zd
Randomly initialize hd ; pd .
repeatInfer hd by using Frank-Wolfe algorithm [40] to maximize

f ðhd Þ ¼

M
X

log

E X
T
N
X
Y
ðpde hdet betwmn Þ

m¼1

e¼1 t¼1

n¼1

Infer pd by using Frank-Wolfe algorithm to maximize
Fig. 16. Streaming MviewLDA result with different batch-sizes.

f ðpd Þ ¼

M
X

log

m¼1

prior knowledge is valuable in streaming learning. SVB will loseprior information while BPS still keep it, therefore, BPS is able toproduce a better result than SVB.
From these experiments, we can see that the appropriate valuesof the scale ratio depend on the quality of the prior knowledge provided into models. Doing a pre-search to find a good range value ofscale ratio will help to improve the result of the learned models.
The result of BPS is better in comparison with SVB in most caseswith the same batchsize settings.

E X
T
N
X
Y
ðpde hdet betwmn Þe¼1 t¼1

n¼1

until convergence
Estimate ~ed , ~zd for each sentence m:
T
X

N
Y

t¼1

n¼1

E
X

N
Y
betwmne¼1

n¼1~edme ¼ Pðem ¼ ejpd ; hd ; wm ; /Þ /~zdmt ¼ Pðzm ¼ tjpd ; hd ; wm ; /Þ /6. Conclusion
In this paper, we discussed the problem of incorporating external knowledge as the prior for streaming Bayesian learning. We
showed that SVB easily forgets prior knowledge as more data come.
This may be problematic when we want to exploit valuable knowledge to deal with the challanges of sparsity and noise. We thenproposed BPS to boost the role of the prior knowledge. Within
BPS, the valuable information from prior is maintained throughlearning process, and hence BPS easily overcomes the vanishingprior problem. BPS provides a simple but effective solution forpractice, and its idea can be easily employed in other streaming
methods.

pde hdet

pde hdet

betwmnAlgorithm 12 Inference of local variables in MviewLDA
Input: Global variable gðlÞ , gðgÞ , document d and label vd
Output: m; l; s; /Randomly initialize m; l; s; /.
repeatm1 ¼ c1 þCRediT authorship contribution statementNX

si ; m2 ¼ c2 þ

i¼1
ðlÞ
lðlÞ
¼ Ijvd ajk þ
jk

Duc Anh Nguyen: Conceptualization, Methodology, Software,
Validation, Formal analysis, Writing - original draft, Visualization,
Investigation. Van Linh Ngo: Conceptualization, Methodology, Validation, Formal analysis, Writing - review & editing, Investigation.
Kim Anh Nguyen: Conceptualization, Methodology, Validation,
Formal analysis, Writing - review & editing, Supervision. Canh
Hao Nguyen: Formal analysis, Writing - review & editing, Supervision. Khoat Than: Conceptualization, Methodology, Software, Validation, Formal analysis, Writing - review & editing, Supervision,
Project administration, Funding acquisition.

siNXð1  si Þ
i¼1

N
X

N
X
ðgÞ
ð1  si Þ/ik

i¼1

i¼1

ðgÞ
ðgÞ
si Ijvd /ðlÞ
þ 1  Ijvd ; lk ¼ ak þ
i;j;k

J X
K
K
X
X
ðlÞ
ðlÞ
ðlÞ
¼ f1 þ expfWðc1 Þ þ Wðc2 Þ 
Ijvd /i;j;k ðWðljk Þ  Wð ljn ÞÞggn¼1

j¼1 k¼1
J X
K X
V
V
X
X
ðlÞ
ðlÞ
ðlÞ
Ijvd /i;j;k Ivwi ðWðgjkv Þ  Wð gjkv ÞÞj¼1 k¼1 v ¼1

v ¼1

K
K
K X
V
V
X
X
X
X
ðgÞ
ðgÞ
ðgÞ
ðgÞ
ðgÞ
ðgÞ
/ik Ivwi ðWðgkv Þ  Wð gkv ÞÞþ /ik ðWðlk Þ  Wð lj ÞÞ þ
k¼1 v ¼1

i¼1

k¼1

ðlÞ
/i;j;k / expfIjvd

si ðWðl

ðlÞ
Þ
jk

1

v ¼1

K
V
V
X
X
X
ðlÞ
ðlÞ
Wð lðlÞ
Ivwi ðWðgjkv Þ  Wð gjkv ÞÞg
j;n Þ þ

v ¼1

n¼1

v ¼1

K
V
V
X
X
X
ðgÞ
ðgÞ
ðgÞ
ðgÞ
ðgÞ
Ivwi ðWðgkv Þ  Wð gkv ÞÞg/ik / expfð1  si ÞðWðlk Þ  Wð lj Þ þ

Declaration of Competing Interest

j¼1

The authors declare that they have no known competing financial interests or personal relationships that could have appeared
to influence the work reported in this paper.

until convergence

157

v ¼1

v ¼1

D.A. Nguyen V.L. Ngo, K.A. Nguyen et al.

Neurocomputing 424 (2021) 143–159[30] C. Lin, Y. He, C. Pedrinaci, J. Domingue, Feature lda: a supervised topic model
for automatic detection of web api documentations from the web,International Semantic Web Conference, Springer. (2012) 328–343.
[31] C. Lin, Y. He, Joint sentiment/topic model for sentiment analysis, in: ACM
Conference on Information and Knowledge Management, 2009, pp. 375–384.
[32] J.G. Ibrahim, M.H. Chen, Power prior distributions for regression models,Statistical Science (2000) 46–60.
[33] J.G. Ibrahim, M.H. Chen, Y. Gwon, F. Chen, The power prior: theory andapplications, Stat. Med. 34 (2015) 3724–3749.
[34] M.C. Hughes, E. Sudderth, Memoized online variational inference for dirichlet
process mixture models, Advances in Neural Information Processing Systems(2013) 1133–1141.
[35] M.I. Jordan, Z. Ghahramani, T.S. Jaakkola, L.K. Saul, An introduction tovariational methods for graphical models, Machine learning 37 (1999) 183–233.
[36] S. Kullback, Information theory and statistics, Courier Corporation, 1997.
[37] J. Turian, L. Ratinov, Y. Bengio, Word representations: a simple and general
method for semi-supervised learning, in, in: Proceedings of the 48th Annual
Meeting of the Association for Computational Linguistics, ACL, 2010, pp. 384–
394.
[38] G.E. Hinton, R.R. Salakhutdinov, Reducing the dimensionality of data withneural networks, science 313 (2006) 504–507.
[39] V. Le, C. Phung, C. Vu, L. Ngo, K. Than, Streaming aspect-sentiment analysis, in:IEEE RIVF International Conference on Computing Communication
Technologies, Research, Innovation, and Vision for the Future (RIVF), 2016,
pp. 181–186.https://doi.org/10.1109/RIVF.2016.7800291.
[40] K.L. Clarkson, Coresets, sparse greedy approximation, and the frank-wolfealgorithm, ACM Trans. Algorithms 6 (2010) 63.
[41] C. Kluckhohn, Human behavior and the principle of least effort. george
kingsley zipf, Am. Anthropol. 52 (1950) 268–270.
[42] P. Xie, E.P. Xing, Integrating document clustering and topic modeling, in:Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial
Intelligence, AUAI Press, 2013, pp. 694–703.

References[1] T. Broderick, N. Boyd, A. Wibisono, A.C. Wilson, M.I. Jordan, Streaming
variational bayes, Adv. Neural Inf. Process. Syst. (2013) 1727–1735.
[2] J. McInerney, R. Ranganath, D.M. Blei, The population posterior and bayesianinference on streams, in: Advances in Neural Information Processing Systems(NIPS), 2015.
[3] A. Masegosa, D.N. Thomas, L. Helge, R. Darío, S. Antonio, L.M. Anders, Bayesianmodels of data streams with hierarchical power priors, in: Proceedings of the34th International Conference on Machine Learning (ICML), 2017, pp. 2334–
2343.
[4] T.D. Bui, C. Nguyen, R.E. Turner, Streaming sparse gaussian process
approximations, Advances in Neural Information Processing Systems (2017)
3299–3307.
[5] M. Faraji, K. Preuschoff, W. Gerstner, Balancing new against old information:
The role of puzzlement surprise in learning, Neural Comput. 30 (2018) 34–83.
[6] Z. Huang, H. Chen, D. Zeng, Applying associative retrieval techniques toalleviate the sparsity problem in collaborative filtering, ACM Transactions on
Information Systems (TOIS) 22 (2004) 116–142.
[7] J. Bobadilla, F. Ortega, A. Hernando, A. Gutiérrez, Recommender systems
survey, Knowl.-Based Syst. 46 (2013) 109–132.
[8] S. Banerjee, K. Ramanathan, A. Gupta, Clustering short texts using wikipedia,
in, in: Proceedings of the 30th annual international ACM SIGIR conference on
Research and development in information retrieval, ACM, 2007, pp. 787–788.
[9] X. Cheng, X. Yan, Y. Lan, J. Guo, Btm: Topic modeling over short texts, IEEE
Trans. Knowl. Data Eng. 26 (2014) 2928–2941.
[10] K. Mai, S. Mai, A. Nguyen, N. Van Linh, K. Than, Enabling hierarchical dirichlet
processes to work better for short texts at large scale, in: Advances in
Knowledge Discovery and Data Mining. Springer. volume 9652 of Lecture
Notes in Computer Science, 2016, pp. 431–442.
[11] J. Tang, Z. Meng, X. Nguyen, Q. Mei, M. Zhang, Understanding the limiting
factors of topic modeling via posterior contraction analysis, in, in: Proceedingsof The 31st International Conference on Machine Learning (ICML), 2014, pp.
190–198.
[12] N. Oppermann, G. Robbers, T.A. Enßlin, Reconstructing signals from noisy datawith unknown signal and noise covariance, Phys. Rev. E 84 (2011), 041118.
[13] L.M. Rickett, N. Pullen, M. Hartley, C. Zipfel, S. Kamoun, J. Baranyi, R.J. Morris,
Incorporating prior knowledge improves detection of differences in bacterial
growth rate, BMC systems biology 9 (2015) 60.
[14] J. Liang, L. Jiang, D. Meng, A. Hauptmann, Leveraging multi-modal priorknowledge for large-scale concept learning in noisy web data, in, in:Proceedings of the 2017 ACM on International Conference on Multimedia
Retrieval, ACM, 2017, pp. 32–40.
[15] B. Luo, Y. Feng, Z. Wang, Z. Zhu, S. Huang, R. Yan, D. Zhao, Learning with noise:enhance distantly supervised relation extraction with dynamic transition
matrix, 2017. arXiv preprint arXiv:1705.03995.
[16] C. Ha, V.D. Tran, V.L. Ngo, K. Than, Eliminating overfitting of probabilistic topic
models on short and noisy text: The role of dropout, Int. J. Approximate
Reasoning 112 (2019) 85–104.
[17] Y. Jo, A.H. Oh, Aspect and sentiment unification model for online review
analysis, ACM International Conference on Web Search and Data Mining(2011) 815–824.
[18] D.M. Blei, A.Y. Ng, M.I. Jordan, Latent dirichlet allocation, Journal of Machine
Learning Research 3 (2003) 993–1022.
[19] L. Theis, M. Hoffman, A trust-region method for stochastic variational
inference with applications to streaming data, in: Bach, F., Blei, D. (Eds.),
Proceedings of the 32nd International Conference on Machine Learning, PMLR,
Lille, France, 2015, pp. 2503–2511. URL:/http://proceedings.mlr.press/v37/
theis15.html.
[20] M.D. Hoffman, D.M. Blei, C. Wang, J.W. Paisley, Stochastic variational inference,
Journal of Machine Learning Research 14 (2013) 1303–1347.
[21] J. Devlin, M.W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep
bidirectional transformers for language understanding, in: Proceedings of the
NAACL-HLT, Association for Computational Linguistics, 2019, pp. 384–394.
[22] D. Erhan, Y. Bengio, A. Courville, P.A. Manzagol, P. Vincent, S. Bengio, Why does
unsupervised pre-training help deep learning?, Journal of Machine Learning
Research 11 (2010) 625–660[23] A. Ahmed, E.P. Xing, Staying informed: supervised and semi-supervised multiview topical analysis of ideological perspective, Empirical Methods in Natural
Language Processing (2010) 1140–1150.
[24] N. Van Linh, N.K. Anh, K. Than, C.N. Dang, An effective and interpretable
method for document classification, Knowl. Inf. Syst. 50 (2017) 763–793,
https://doi.org/10.1007/s10115-016-0956-6.
[25] G.A. Diamond, S. Kaul, Prior convictions: Bayesian approaches to the analysis
and interpretation of clinical megatrials, J. Am. Coll. Cardiol. 43 (2004) 1929–
1939.
[26] M.E. Alfaro, M.T. Holder, The posterior and the prior in bayesian phylogenetics,
Annu. Rev. Ecol. Evol. Syst. 37 (2006) 19–42.
[27] Newman, E. Mark, Power laws, pareto distributions and zipf’s law, Contemp.
Phys. 46 (2005) 323–351.
[28] Piantadosi, T. Steven, Zipfs word frequency law in natural language: a criticalreview and future directions, Psychonomic Bull. Rev. 21 (2014) 1112–1130.
[29] I. Sato, H. Nakagawa, Topic models with power-law using pitman-yor process,
ACM SIGKDD International Conference on Knowledge Discovery and Data
Mining (2010) 673–682.

Duc Anh Nguyen graduated from Hanoi University of
Science and Technology with a master degree in Information Technology. He is currently a Ph.D student at
Kyoto University, Bioinformatics Center. His interest
focuses on representation learning and its application.

Van Linh Ngo is currently a PhD student at Hanoi
University of Science and Technology (HUST), Vietnam.
He also received B.S (2011) and M.S (2014) degrees from
HUST. He is a member of Data Science Laboratory, HUST.
His research interests include topic model, continual
learning, recommender systems, and big data.

Kim Anh Nguyen received B.S (1988) and Ph.D. (1994)
from Hanoi University of Science and Technology. Her
recent research interests include topic modeling, opinion mining, deep learning, graph analytics with big data.

158

Neurocomputing 424 (2021) 143–159

D.A. Nguyen V.L. Ngo, K.A. Nguyen et al.
Canh Hao Nguyen received his B.S. degree in Computer
Science from the University of New South Wales, Australia, M.S. and Ph.D. degrees from JAIST, Japan. He has
been working in machine learning and bioinformatics.
His current interests are machine learning for graphdata, sparse modeling with applications in biologicalnetwork analysis.

Khoat Than is currently an associate professor at Hanoi
University of Science and Technology. He received Ph.D.
degree from Japan Advanced Institute of Science andTechnology in 2013. He joins the Program Committees
of various leading international conferences, including
ICML, NIPS, IJCAI, ICLR, PAKDD, ACML. His recent
research interests include representation learning, deep
generative models, topic modeling, continual learning.

159