Exploring Time-Sensitive Variational BayesianInference LDA for Social Media Data
Anjie Fang1 , Craig Macdonald2 , Iadh Ounis2 , Philip Habel2 , Xiao Yang2
1

a.fang.1@research.gla.ac.uk,2 {firstname.secondname}@glasgow.ac.uk
University of Glasgow, Glasgow, UK

Abstract. There is considerable interest among both researchers andthe mass public in understanding the topics of discussion on social media as they occur over time. Scholars have thoroughly analysed samplingbased topic modelling approaches for various text corpora including social media; however, another LDA topic modelling implementation—
Variational Bayesian (VB)—has not been well studied, despite its known
efficiency and its adaptability to the volume and dynamics of social media data. In this paper, we examine the performance of the VB-based
topic modelling approach for producing coherent topics, and further, we
extend the VB approach by proposing a novel time-sensitive Variational
Bayesian implementation, denoted as TVB. Our newly proposed TVB
approach incorporates time so as to increase the quality of the generated
topics. Using a Twitter dataset covering 8 events, our empirical results
show that the coherence of the topics in our TVB model is improvedby the integration of time. In particular, through a user study, we findthat our TVB approach generates less mixed topics than state-of-the-art
topic modelling approaches. Moreover, our proposed TVB approach canmore accurately estimate topical trends, making it particularly suitable
to assist end-users in tracking emerging topics on social media.

1

Introduction

Perhaps the greatest technological change over the past decade has been theadvent and growth of social media. Yet despite social media’s ubiquity, scholars
still wrestle with the appropriate tools for best capturing the topics of discussion
conveyed over these platforms [1–3]. To this end, researchers have employed various topic modelling approaches [1, 4–8], e.g. Latent Dirichlet Allocation (LDA),
but these e↵orts have proved challenging, as models applied to social media data
can produce topics that are mixed and lack coherence, and are generally difficult to interpret [1]. To deal with the short nature of social media posts, LDA
enhancement methods such as single topic assignment [1, 9, 10] and document
pooling [2, 11] have been proposed to improve the coherence of the generated
topics within the sampling-based topic modelling approaches. However, another
LDA implementation, the Variational Bayesian (VB)-based topic modelling approach, has not been well studied on social media posts. As the VB approach has
been shown to be more efficient for large datasets [6, 12], it can be argued that
VB can better handle the increasing volume and dynamicity of social media data.

It has been previously shown that the time dimension of documents (e.g.
news articles) can help a topic modelling approach to provide more valuable
information [7, 8, 13], for example, capturing the topic changes or topical trends
over time. Apart from these additional benefits, we argue that distinguishing
topical word usage over time can also help to generate more coherent and lessmixed topics, thereby assisting the end-users in interpreting discussions on social
media. We propose a time-sensitive VB (TVB) approach for social media datathat embraces the time dimension of social media data. We extend the traditional VB approach by incorporating a Beta distribution, which is reported to fit
various patterns [14]. The employed Beta continuous distribution is used to represent each topic’s volume over time, i.e. the topical trend, similar to what has
been used in [7]. However, we notice that time could have a negative bias on the
topic inference when a Beta distribution does not fit the topics’ trends. To solve
this problem, we introduce a balance parameter to alleviate the bias of time.
To evaluate the performance of the proposed TVB approach, we create aground truth Twitter dataset covering 8 large events. We evaluate our TVB
approach together with several baselines from the literature (e.g. Twitter LDA(TLDA) [1], the Topic Over Time approach (TOT) [7]) in terms of topical coherence, the extent to which the generated topics are mixed, or the estimation errorsof the topical trends. Our empirical results suggest that incorporating the time
dimension does indeed help to enhance the coherence of the topics generated by
the TVB approach compared with the traditional VB and sampling approaches.
Moreover, we show that our TVB model can outperform the state-of-the-art
LDA enhancement approaches (i.e. TLDA and TOT) in generating less mixed
topics. This conclusion is further supported by conducting a user study to validate the results of the quantitative evaluation. Finally, we compare our TVB
approach with the TOT approach when estimating the topical trends. We findthat our proposed TVB model better estimates the topical trends.
The contributions of this paper are three-fold: 1) we study the VB approachand develop its enhancement for social media, 2) we propose a time-sensitive
TVB approach by integrating the time dimension in the modelling process and3) we show the advantages of the TVB approach in generating better quality
topics and estimating more accurate topical trends.
The rest of this paper is organised as follow: Section 2 provides basic background on two LDA implementations, i.e. sampling & VB approaches, followed
by a description of related work in Section 3. We describe our TVB approachin Section 4. Following that, we describe our dataset in Section 5 and the experimental setup in Section 6. The results are shown and discussed in Section 7.
Finally, we provide concluding remarks in Section 8.

2

Two LDA implementations: Sampling & VB approaches

In topic modelling approaches, a topic k is represented by a distribution k (kis the topic index and K is the number of topics) over N terms drawn from a
Dirichlet prior ⌘, where N is the size of the vocabulary. A document in a corpus w
is represented by wd = {wd,1 , ..., wd,i , ..., wd,N } (d is the document index and D isthe number of documents in w) and has a topic belief distribution ✓d drawn fromthe Dirichlet prior ↵. A document wd is associated with topic assignment zd ={zd,1 , ..., zd,i , ..., zd,N }. The sampling approach [4, 5], which is based on a Markov
Chain Monte Carlo sampling, estimates the real posterior distributions (e.g. k& ✓d ). In a typical sampling approach, such as the collapsed Gibbs sampling,each word is assigned a topic according to Equation (1) in order to construct aw
Markov Chain on latent topics, where n d,i(d,i),k is the number of wd,i occurring indtopic k and n (d,i),j is the number of words from document wd occurring in topic
k not including the current one. After a number of iterations,({ 1 , .., K })and ✓ ({✓1 , .., ✓D }) can be estimated from the converged Markov Chain.
n

wd,i

+⌘

(1)

(d,i),kdp(zd,i = k|z (d,i) , w) = n (d,i),k+N ⌘ ⇥ (n (d,i),j + ↵)The VB approach [6, 12] approximates the variational distribution by minimising the distance from the true distribution. Specifically, an expectation
maximization (EM) algorithm is used to maximise the lower bound of the loglikelihood of all documents, which equivalently minimises the distance between
the variational distribution and the true posterior distribution. In the E step ofEM, the variational Dirichlet prior d of all documents are optimised together
with D⇥N ⇥K , which represents the words’ topic belief within documents. In
the M step of EM, D⇥N ⇥K is used to update the variational Dirichlet prior. The parameters’ optimisation formulas in the EM algorithm areK⇥N ofshown in Equation (2). Finally, and ✓ can be obtained when the lower boundconverges. Importantly, since the VB approach does not have the topic assignment step, the single topic assignment strategy (mentioned in the introduction
and discussed further in Section 3) cannot be applied. The main advantage ofthe VB approach is that the lower bound converges much more quickly than
the sampling approach especially on large datasets [6, 12]. Moreover, the VB
approach can be intuitively implemented in parallel since the updates of d &d among documents do not impact each other, while the sampling approachcannot be easily parallelised as it is intrinsically sequential [15]. Because of theincreasing volume of social media data and its dynamicity, it could be argued
that the VB approach o↵ers various advantages for those interested in interpreting discussions on social media as events transpire. In the next section, we
review a number of existing methods, which aim to improve the quality of topic
models and/or integrate the time dimension in the topic model.
d,i,k / exp{E[log k,i ] + E[log✓d,k ]},3d,k = ↵ +Pi,kd,i,k ,

k,i = ⌘ +

P

d,i,k

d,i,k

(2)

Related Work

Three methods are mainly used in the literature to adapt topic modelling approach for short social media posts: 1) A post is assigned to a single topic under
the assumption that a post represents a single topic. This method was used in
Twitter LDA as proposed by Zhao et al. [1] and later applied in [9, 10]. Indeed,this method brings more coherent words for a topic. However, we argue that thismethod can generate multi-theme topics1 since the underlying assumption cannot be upheld in all situations. For example, the same words can be used across1

A mixed topic contains keywords pertaining to multiple di↵erent topic themes.

multiple topics. Assigning all words in a tweet to a single topic could increase
word overlaps and thus result in mixed topics. 2) Multiple posts are combined
into a virtual document [2, 11], also known as the pooling strategy (e.g. tweets
from a single user are combined into a single document [2]). The pooling method
can increase the number and occurrence of words, which makes it easier to apply
a topic modelling approach. 3) Topical words are connected using word representations (e.g. word embedding). Sridhar et al. [16] improved the topical coherence
by applying soft clustering over word representations in a topic model. Nguyen etal. [17] introduced an additional word topic belief distribution calculated using
word representations in the sampling approaches. Li et al. [18] assigned the semantically similar words under the same topic. All of these approaches improved
the topical coherence by connecting similar words in order to overcome the shortness of posts on social media. We do not deploy the single topic assignment
method in our approach since it cannot be applied in the VB approach, as mentioned in Section 2. Given that the central aim of this paper is to integrate time tothe VB approach, we do not adopt the pooling method in our modelling process.
Early work on time-sensitive topic modelling by Blei et al. [13] was based on a
Markov assumption that the topic parameters are in a sequential structure overtime. Later on, Blei et al. [19] used Brownian motion to estimate the topical evolutions over time. The proposed model was claimed to have a better predictive
perplexity. However, these state-space models did not integrate the timestampsof documents in the generative process. Assuming that the topic proportion
changes over time, Wang et al. [7] proposed a non-Markov topic model (TOT)
using a Beta continuous distribution, which was reported to generate more interpretable topics and trends. Their work is based on a sampling approach, inwhich the timestamps of documents are incorporated in the generative process
without considering time dependency for topics or words. Another recent workfrom [8] leveraged a time-dependent function to capture topical dynamics.
Although the sampling approach is still the preferred choice in analysing social media data, the advantages of the VB approach for a large corpus should
not be ignored. For example, Ho↵man et al. [20] and Braun et al. [12] recently
proposed a VB-based solution to quickly inference a large number of documents.
In this paper, we o↵er a solution to apply an enhanced VB approach (TVB)
for social media data, which incorporates time in the topic modelling process.
Our TVB approach is based on the same assumption as the TOT approach but
is implemented using VB. In the next section, we introduce our TVB approachand elaborate further the di↵erences between the TOT and TVB approaches.

4

Integrating the time dimension in the VB Approach

Our proposed TVB approach extends the traditional VB approach by integrating the time dimension of social media data. In this section, we explain how we
implement the EM algorithm in our proposed TVB approach and compare itwith the traditional VB and TOT approaches. To integrate the timestamps ofsocial media posts, we deploy a continuous probability distribution ⌧ for each
topic. This time distribution ⌧k represents the proportion of topic k over time.
Theoretically, any continuous distribution can be used to simulate the topic

proportion over time. However, to better estimate topical trends, the continuous
distribution has to approximate the real topical trends. Indeed, recently, the Beta
distribution has drawn a lot of attentions for accommodating a variety of shapes
given an x-axis interval [14]. Therefore, we choose to use a Beta distribution since
it can more accurately fit the various shapes of topical trends. Next, we describe
the generative process and the EM implementation of our TVB approach.
Generative process. Similar to the traditional LDA generative process,each word wd,i in a document d is assigned a topic assignment zd,i according
to ✓d , where i is the word index. Since words (w) in social media posts areassociated with timestamps (t), in the TVB approach, a pair (wd,i , td,i ) is drawn
from zd,i and ⌧zd,i , respectively, where a Beta distribution ⌧k is parametrisedby two shape parameters, ⇢1k and ⇢2k . A similar strategy was previously appliedin a time-sensitive sampling approach [7]. The process is defined as follows:
zd,i |✓d ⇠ Dirichlet(↵),

wd,i |zd,i ,

td,i |zd,i , ⌧zd,i ⇠ Beta(⇢1zd,i , ⇢2zd,i )

zd,i ⇠ Dirichlet(⌘),

EM Implementation of the TVB approach. The core part of a variational inference is to minimise the distance between the variational distributionsq(✓d | d ) & q( k | k ) and the two true posterior distributions p(✓d |↵) & p( k |⌘),
i.e. maximising the lower bound of a document log-likelihood p(w, t|↵, ⌘) shownin Equation (3). The right part of the equation is the lower bound of all documents, L. Commonly, the derivative of L is taken over parameters ( , , ) andthus the parameter optimisation formulas can be obtained by maximising thelower bound L. To achieve this, we first decompose all the items in L.
=P

log p(w, t|↵, ⌘)L(w, t, , ) =P

d Eq [log p(wd , td , zd , ✓d ,, ⌧ |↵, ⌘)])] + Eq [log p(zd |✓d )] + Eq [log p(✓d |↵)] + Eq [log p( |⌘)] (3)d (Eq [log p(wd |zd ,+Eq [log p(td |zd , ⌧ )]Eq [q(✓d , zd , )])The sixth item of the lower bound L, the log-expectation of joint variational
probability, is decomposed as shown in Equation (4). These decomposed items together with the first five items in L can be expanded by leveraging the propertiesof the Dirichlet and Beta distributions. Finally, we have the expanded L shownin Equation (5), where B is the Beta function and is the Gamma function.
Eq [q(✓d , zd , )] =Pk Eq [log q(✓d,k | d,k )] +L(w, t, , ) =

P P
d(
i,k

+log (K↵)+

P

P+log (
i,ki,k1
d,i,k ((⇢k

P

Klog (↵) +Pi,k ⌘)

d,i,k log d,i,k

P

k

k) +

log (

P

i Eq [log q(zd,i | i,k )] +d,i,k Eq [logi,k log1)log td,i + (⇢2k
log (

P

i,k

P

(⌘) +

1)log (1Pk logk,i ) +( k)Pk,i ] +

k (↵

P

P

i,k (⌘i,ki,k logk( ki,k Eq [log q( k,i | k,i )](4)d,i,k Eq [log ✓d,k ]1)Eq [log✓d,k ]1)Eq [logtd,i ))

P

P

P P
k(
i

k,i ]

1
2
d,i,k log B(⇢k , ⇢k ))(5)1)Eq [log✓d,k ]( k,i )

Pi,k ( k,i@L d,i,k@ d,i,k1)Eq [logk,i ])

To maximise L, we first optimise d,i,k by setting= 0 and obtain
the d,i,k optimisation formula shown in Equation (6). Compared with the traditional VB approach, the third item in Equation (6), the time statistics, is the
additional feature we add to incorporate timestamps. Intuitively, the time statistics can have a direct impact on the term topic belief d,i,k . If a word wd,i isAlgorithm 1: Our TVB approach for Latent Dirichlet Allocation.
Initialize

N ⇥K , D⇥K
while L not converges do
E step:for d < D do
repeatfor i < N d & k < K do
d,i,k / exp(Eq [log k,i ] + Eq [log✓d,k ]+ ((⇢11)log td,i + (⇢21)log (1td,i )kkPd,k = ↵ +i,k d,i,k
until

d converges;

M step:(⇢1
k)

2
log B(⇢1k , ⇢k )))(⇢1
kk,i = ⌘ +

P

d,i,k

P

d,i,k d,i,k log td,i
, (⇢2
P
k)
d,i,k d,i,k
d,i,k , 8i 2 N

⇢2
k) =

(⇢1
k

⇢2
k) =

P

d,i,k d,i,k log (1Pd,i,k d,i,k

td,i )

highly used in topic k at a time point t, d,i,k is likely to be promoted if a post
has the word wd,i with a timestamp t. However, the estimated time distribution
may not always fit a topic’s trend well. A drifted time distribution could give anegative bias on d,i,k . To solve this problem, we introduce a balance parameter, to control the impact of the time statistics on d,i,k and alleviate such bias.
Note that the influence of time in the TOT approach cannot be adjusted, e.g.
through a parameter. Similar to d,i,k , we obtain the optimisation formula ofand (shown in Equation (7)) by setting their derivative of L to zero.
d,i,k / exp(Eq [log k,i ] + Eq [log✓d,k ] +d,i = ↵ +

P

((⇢1k
i,k

1)log td,i + (⇢2kd,i,k ,

k,i = ⌘ +1)log (1Pd,i,ktd,i )

log B(⇢1k , ⇢2k )))

(6)
(7)

d,i,k

Meanwhile, to maximise L, we can also take the partial derivative with respect to the parameters of Beta distribution, ⇢1k /⇢2k . Actually, this step is equivalent to maximising the likelihood of the timestamps in topics. By optimising⇢1k /⇢2k , we also obtain the estimated topical trends. Taking the derivative tozero, we obtain the optimisation formula of ⇢1k /⇢2k shown in Equation (8). Since
the Digamma function ( , log-derivative of ) is involved in the optimisation
equation, it is impossible to calculate ⇢1k /⇢2k directly. In our TVB approach, weestimate ⇢1k /⇢2k using a parameter optimisation algorithm and we set their initialvalues following [21]. Note that, while we use EM to estimate ⇢1k /⇢2k , the methodof moment [7] is used in the TOT approach. In summary, in the iterative EM
algorithm, we update and for each document (social media post) in the E
step. In the M step, and ⇢1k /⇢2k are updated using the statistics information( ) from all posts. At the same time, all the timestamps are taken into accountto estimate ⇢1k /⇢2k . Algorithm 1 shows the EM algorithm in our TVB approach.
(⇢1k )

5(⇢1k⇢2k ) =

P

d,i,k

P

d,i,k log td,i

d,i,k

,

d,i,k

(⇢2k )(⇢1k⇢2k ) =

Pd,i,kd,i,k log (1

P

d,i,k

d,i,k

td,i )(8)

Ground Truth Datasets

To evaluate our proposed TVB approach together with the existing topic modelling approaches, we create a Twitter dataset containing 8 selected popular
hashtag-events that occurred in July and August 2016. This dataset was collected using Twitter API by searching for 8 hashtags: #gopconvention, #teamgb,

#badminton, #gameofthrone, #juno, #nba, #pokemongo and #theresamay. For
each hashtag-event, we randomly sample 2,000 tweets, hence we obtain a Twitter dataset containing 16,000 tweets. Such a balanced dataset has several advantages: 1) The reasonable size (16K) of the Twitter corpus allows for the efficient
conduct of our experiments, i.e. all approaches can quickly converge; 2) We avoid
generating dominant and duplicated topics, thereby focusing the evaluation on
the quality and coherence of the topics; 3) These predefined hashtags provide
readily usable ground-truth labels, i.e. each hashtag-event is associated with the
top 10 used words in its corresponding tweets. These labels of the 8 hashtagevents are used to match a generated topic with a hashtag-event. This enables us
to evaluate how close the estimated topical trend to its real trend (further details
are given in Section 6); 4) This ground truth dataset allows humans to more e↵ectively examine the generated topics and to conduct a user study described in Section 7. Indeed, since this dataset contains a limited number of topics, it is morefeasible for human interpreters to evaluate all the generated topics of a given
topic model in the conducted user study. In the next section, we explain how weapply various topic modelling approaches on this dataset and the used metrics.

6

Experimental Setup

We compare our new proposed TVB approach to 4 baselines from the literature,
namely TOT [7], TLDA [1], and the traditional sampling (Gibbs) [4] and VB [6]approaches. In particular, the TOT approach is included since it is the mostclosely related work that integrates the time dimension into the topic modelling
process. We use 3 di↵erent metrics to evaluate the quality of the generated topic
models: 1) the topical coherence, 2) the degree to which the topics are mixed and3) the topical trends estimation error. In the following, we explain the experimental setup used for the topic modelling approaches and each of the metrics.
Topic Modelling Setup. For all approaches (Gibbs, TLDA, TOT, VB &TVB), ⌘ is set to 0.01 according to [4, 5]. We do not follow the traditional settingfor ↵ (↵ = 50/K), and set instead ↵ to 0.4 for all approaches in our experiments,
since in other separate preliminary experiments we noticed that a smaller ↵ helps
to generate topics with higher coherence for short texts. The number of topics is
set to 10, which is slightly higher than the real number of topic (8 in our dataset
corresponding to 8 hashtags) because a slightly higher number of topics assuresthat all hashtag-events can be extracted. As our Twitter dataset is not very big
and contains distinguishable topics, all approaches can converge fast. Hence, forthe sampling approaches (Gibbs, TLDA and TOT), we set the maximum numberof iterations to 50. For the traditional VB and our proposed TVB approaches, we
set the number of iterations to 10 as the VB approaches converge more quickly.
Each experiment for each approach is repeated 10 times in order to conduct
statistical significance. In TLDA, a document contains several tweets from a single Twitter user. However, most of users in our Twitter dataset have only one
tweet. Hence, we create a virtual Twitter user by assigning 5 random tweets tothis user. For all the other approaches, a document represents a single tweet.
Metric 1: Coherence Metric. A coherence metric is used to evaluate
whether a generated topic is interpretable by humans. A higher score indicatesthat the topic is easier to understand. Following [22, 23], we use a word embedding (WE) representations-based coherence metric to evaluate the coherence ofthe generated topics, which has been reported to have a high agreement with human judgments. In order to capture the semantic similarity of the latest hashtags
and Twitter handle names, we train our WE model using 200 million English
tweets posted from 08/2015 to 08/2016. The obtained WE model has 5 million tokens. We use the average coherence (Aver) to evaluate all topics in a topic
model. Meanwhile, we also examine the top 2/72 most coherent topics in a model
for more e↵ective coherence evaluation, i.e. C@2 & C@7 metrics, following to [24].
Metric 2: Topics Mixing Degree. A generated topic can be a mixture ofseveral topic themes (multi-theme topics). The coherence score is calculated byaveraging the similarity of each two words in the top 10 words of a generated
topic. Consider that if a topic contains two topic themes, as long as the coherenceof words under a theme in this topic is high, the coherence metric can still yielda higher coherence. Although this multi-theme topic is interpretable by humans,
a user expects to see the generated topics only containing a single topic theme.
Therefore, it is necessary to identify the multi-theme topics. Since the generated
multi-theme topics often contain the same topic theme, these multi-theme topics
can be similar to each other. Thus, to quantify the extent to which a given topic
in a model is mixed (MD), we use Equation (9) to calculate the topic similarity
in the entire topic model (containing K topics). The higher the similarity, themore likely that the model has more multi-theme topics. A similar methodology
is used in [25] to identify the background topics.
M D( ) =P P
kk0 cosine( k ,k0 )/||2(9)

Metric 3: Topical Trends Estimate Error. Both the TOT and our TVB
approaches estimate the topical trends. To evaluate the topical trends over time,
we calculate the distance/error between the real topic trends and the estimatedtopical trends (using the Beta distributions in the TOT/TVB models). The error
is calculated using the method shown in Equation (10), where P DFk (t) is theprobability density of the real timestamps of topics, which is obtained throughthe ground truth Twitter dataset. The ERR score ranges from 0 to 2. The generated topics are matched to the ground-truth topics if the top 10 words of a generated topic have at least 33 same words in the top 10 words of a hashtag event.
ERR(⌧ ) =7

Results

P R1
k0|⌧k (t) P DFk (t)|dt
K(10)

In this section, we analyse our experimental results shown in Table 1. The listed
scores are the average scores of 10 models generated by each approach with respect to the 3 types of metrics (described in Section 6). For the coherence metrics(Aver, C@2 & C@7), a higher score means more coherent topics, whereas lower
scores for the MD and ERR metrics indicate higher quality models. The subscript
indicates whether a given approach is significantly4 better than the other one.
2

Considering that the number of topics is 10, the top 2 and 7 most coherent topics arereasonable choices for a comprehensive coherence evaluation. 3 3 mutual words in the
top 10 words is a reasonable minimum number to indicate a similar topic. 4 p-values(p<0.05) are calculated by the t-test using 10 models of each two approaches.

Table 1. The topic coherence, mixing degree and topic trends estimation error.
ModelsGibbs (G)TLDA (W )VB (V )TOT (T )TVB(T 0 ),TVB(T 0 ),TVB(T 0 ),TVB(T 0 ),

= 0.4
= 0.6
= 0.8
= 1.0

Aver

Coherence
C@2

C@7

0.154
0.177G,V,T ,T 0
0.151
0.160G,V

0.204
0.248G,V,T ,T 0
0.201
0.205

0.152
0.153
0.158V
0.156V

0.202
0.204
0.221G,V,T
0.209

MD

ERR

0.168
0.198G,V,T ,T 00.165
0.175V

0.051W,T
0.102T
0.049W,T
0.149

⇥
⇥
⇥
1.358

0.165
0.166
0.174V
0.170

0.043W,T
0.042W,T
0.047W,T
0.055W,T

1.211T
1.256T
1.206T
1.168T

Fig. 1. The real and estimated topical trends, where x-axis and y-axis represent the
timeline and density probability, respectively.

For example, the average coherence score (Aver) of TVB =0.8 ( is the balanceparameter) is significantly better than that of the VB approach, indicating that
TVB generates topics with higher coherence than VB. To help understand the
topical trends, we randomly choose one TOT and one TVB models and list theirestimated topical trends together with the real trends in Figure 1. Next, we will
first analyse the results in terms of topical coherence and topical mixing degree.
Then, we discuss the performances of the approaches in estimating topical trends.
Topical Coherence and Topical Mixing Degree. Our experiments involve two types of topic modelling approaches: Sampling & VB approaches(shown as G, W, T & V, T 0 in Table 1) and two topic enhanced methods: single topic assignment (W ) and incorporation of the time dimension (T, T 0 ). First,
for the topical coherence, it is clear that the single topic assignment, TLDA (W ),significantly outperforms all of the other approaches. However, we can still see
the positive impact of the time dimension in improving the coherence of models in both the TVB and TOT approaches. For example, the Aver coherence
score of TOT is better than that of the Gibbs and VB approaches. In particular, for C@2, the TVB models outperform all of the other approaches, except
the TLDA models with = 0.8, while the TVB models with a lower/higher(T 0 with = 0.4, 0.6/1.0) do not. This indicates that alleviating the bias of thetime statistics (described in Section 4) helps to generate topics with a higher
coherence. In terms of the MD metric, the TLDA models have higher mixing
scores indicating they have more multi-theme topics. As argued in Section 3, aggressively assigning all words in a tweet under the same topic theme can result

Table 2. Topic samples from TLDA and TVB =0.8 models, where the underlined words
have a di↵erent topic theme from the others in a topic. Note that, we present a humanwith the top 10 words of a topic in our user study. We list the top 5 words for each
topic in this table due to space limitations.
Topic

TLDA

TVB =0.8

1
2
3
4
5
6
7
8

#rio #badminton #olympics #iamteamgb wei#jupiter #juno @nasa orbit @nasajuno#nbasummer nba #basketball @nba basketball@gameofthrones #emmys season outstanding#rncincle trump speech melania donald#rio #badminton #iamteamgb team gold
#iamteamgb #theresamaypm thanks #jupiterthrones game pokemon season like #pokemon#badminton #rio #mas #olympics wei chong#juno burn engine complete unlock #jupiternba #basketball sign wire basketball
thanks @gameofthrones #iamteamgb #emmys#rncincle trump @realdonaldtrump speech#iamteamgb win medal #rio @teamgb#theresamaypm watch #brexit minister prime
pokemon basketball team usa #pokemon news

in multi-theme topics. The MD results confirm that this single topic assignment
indeed causes multi-theme topics, which is the reason why the TLDA models
exhibit a very high mixing degree. Besides, we notice that the TOT models havethe highest topical mixing degree. This might be caused by the strong time bias
in the sampling approach. Consider that if two topics have similar trends (topical
proportions over time, e.g. #nba and #pokemongo in Figure 1), it is likely thatthese two topics would mix, and thus it causes the generation of multi-theme
topics in the TOT models. In this situation, reducing the importance of the time
statistics by the balance parameter is equally increasing the importance of thewords statistics (the first two items in Equation 6), hence avoiding the negative
bias of time statistics.
To verify that our generated TVB models have less multi-theme topics than
TOT & TLDA, we also conduct a user study to compare the mixing degree of
their generated topic models. Since the MD scores of the TOT models are significantly higher than those of the TLDA models, we choose to compare themixing degree between the TVB =0.8 and TLDA models using human judgements. If the users confirm that the TVB approach generates less multi-theme
topics than TLDA, it is reasonable to conclude that the TVB approach generates
less multi-theme topics than TOT. In our user study, we ask 8 expert end-users
whether a given topic contains multiple themes. Specifically, both the TVB and
TLDA approaches generate 10 models. We pair these 20 models randomly andgenerate 10 pairs, where each pair has one model from TVB and another one
from TLDA. For each pair, we present a human with all the generated topicsof the 2 models. The human is asked to identify all of the multi-theme topics from 2 given models (10 topics per model). A model in a pair is preferred(i.e. obtains a vote), if a human finds less multi-theme topics in this model
pair. Each pair gets 3 judgements from 3 di↵erent humans. An approach getsa credit if its model in a pair obtains the majority votes out of 3. In the end,
among the 10 pairs, our TVB approach gets 7 credits while the TLDA approachgets 2 credits, expect that 3 humans do not have agreement on one pair outof the 10. This user study confirms the results we obtained from MD that our
TVB approach generates less multi-theme topics. We list two topic examplesof our TLDA and TVB =0.8 models in Table 2. Both models generate humaninterpretable topics. However, we can see more multi-theme topics in the TLDA
models, such as “badminton”(topic 1), “teamgb” (topic 6), “theresamaypm”(topic 7) and “pokemon”(topic 8), while the TVB model has less multi-theme
topics: “gameofthrone” (topic 4) and “pokemon”(topic 8). In fact, it is easy tomix the topics “theresamaypm” and “teamgb” since they are all popular topics
in the UK, and it is possible that the word usage in these two topics is similar.
However, the topical trends of these two topics are not similar: “theresamaypm”
was popular around 11/07/2016 when Theresa May became the new UK Prime
Minister, while “teamgb” was highly discussed during the Olympic Games (from05/08/2016 to 21/08/2016) (See the topical trends in Figure 1). Our TVB approach can identify these di↵erent topical trends by integrating time.
Topical Trends Estimation Error. Both the TOT and TVB approaches
estimate topical trends. The ERR metric indicates the distance between the real
topical trends and the estimated ones (smaller distances are better). The ERR
scores in Table 1 suggest that our TVB approach generates significantly moreaccurate topical trends than the TOT approach. The main reason is that the
TOT approach has a very high mixing degree (see Table 1), which shows that it
has more multi-theme topics similar to the TLDA approach. It could be difficult
to match the real topics with the generated topics (explained in Section 6), andthus the multi-theme topics in the TOT model result in less accurate topicaltrends. Unlike the TOT/TLDA approaches, our TVB model has less multitheme topics, which results in a more accurate estimation of the topical trends.
In Figure 1, both chosen models have duplicated topics, which are #badminton &#juno and #gameofthrone & #juno in the TOT and TVB models, respectively.
Since the TOT models have more multi-theme topics, it is difficult to matchthe generated topics with the real ones. For example, the topic theme #nba ismixed with #pokemongo in the TOT model. As a result, the estimated trend of
TOT for #nba is not accurate. Although both the TOT and TVB models do notexactly fit the real topical trends using Beta distributions, it is still clear thatthe estimated trends from the TVB model are closer to the real trends thanthose of the TOT model as illustrated in Figure 1.
Apart from the used three metrics, it is worth recalling that all the VB andTVB models in our experiments are obtained by setting the maximum iteration
to 10, while it is set to 50 for the TLDA, TOT & Gibbs models (see Section 6).
Using less iterations, our TVB approach can still provide very competitive results, which indicates its advantage in terms of convergence speed.

8Conclusions

In this paper, we proposed a time-sensitive Variational Bayesian (TVB) topic
modelling approach to improve the quality of generated topics and to estimate
topical trends by leveraging the time dimension of social media posts. Our proposed TVB approach, extending the traditional Variational Bayesian approach,employed a Beta distribution to integrate time, where the time statistics were
controlled by a balance parameter to alleviate bias. Through experimentation
over a ground truth Twitter dataset covering 8 hashtag events, we showed thatthe time dimension helps to generate more coherent topics in our models withthe set balance parameter. Backed by a user study, we find that our TVB approach generated less mixed topics compared with two state-of-the-art baselines.
Moreover, our TVB approach can more accurately estimate the topical trendsof social media posts.

References
1. Zhao, W.X., Jiang, J., Weng, J., He, J., Lim, E.P., Yan, H., Li, X.: Comparing
Twitter and traditional media using topic models. In: Proc. of ECIR. (2011)
2. Mehrotra, R., Sanner, S., Buntine, W., Xie, L.: Improving LDA topic models for
microblogs via tweet pooling and automatic labeling. In: Proc. of SIGIR. (2013)
3. Fang, A., Ounis, I., Habel, P., Macdonald, C., Limsopatham, N.: Topic-centricclassification of Twitter user’s political orientation. In: Proc. of SIGIR. (2015)
4. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. the Journal ofmachine Learning research 3 (2003) 993–1022
5. Griffiths, T.L., Steyvers, M.: Finding scientific topics. Proceedings of the National
Academy of Sciences 101 (2004) 5228–5235
6. Blei, D.M., Jordan, M.I.: Variational methods for the dirichlet process. In: Proc.
of ICML. (2004)
7. Wang, X., McCallum, A.: Topics over time: a non-markov continuous-time modelof topical trends. In: Proc. of SIGKDD. (2006)
8. Hong, L., Dom, B., Gurumurthy, S., Tsioutsiouliklis, K.: A time-dependent topic
model for multiple text streams. In: Proc. of SIGKDD. (2011)
9. Cheng, X., Yan, X., Lan, Y., Guo, J.: Btm: Topic modeling over short texts. In:
Proc. of TKDE. (2014)
10. Yan, X., Guo, J., Lan, Y., Xu, J., Cheng, X.: A probabilistic model for bursty
topic discovery in microblogs. In: Proc. of AAAI. (2015)
11. Weng, J., Lim, E.P., Jiang, J., He, Q.: Twitterrank: finding topic-sensitive influential twitterers. In: Proc. of ICWSM. (2010)12. Braun, M., McAuli↵e, J.: Variational inference for large-scale models of discrete
choice. Journal of the American Statistical Association 105 (2010) 324–335
13. Blei, D.M., La↵erty, J.D.: Dynamic topic models. In: Proc. of ICML. (2006)
14. Guolo, A., Varin, C., et al.: Beta regression for time series analysis of bounded
data. The Annals of Applied Statistics 8 (2014) 74–88
15. Asuncion, A., Welling, M., Smyth, P., Teh, Y.W.: On smoothing and inference for
topic models. In: Proc. of CUAI. (2009) 27–34
16. Sridhar, V.K.R.: Unsupervised topic modeling for short texts using distributed
representations of words. In: Proc. of NAACL-HLT. (2015)17. Nguyen, D.Q., Billingsley, R., Du, L., Johnson, M.: Improving topic models withlatent feature word representations. In: Proc. of TACL. (2015)
18. Li, C., Wang, H., Zhang, Z., Sun, A., Ma, Z.: Topic modeling for short texts with
auxiliary word embeddings. In: Proc. of SIGIR. (2016)
19. Wang, C., Blei, D., Heckerman, D.: Continuous time dynamic topic models. In:
Proc. of CUAI. (2008)
20. Ho↵man, M., Bach, F.R., Blei, D.M.: Online learning for latent dirichlet allocation.
In: Proc. of NIPS. (2010)
21. Johnson, N.L., Kotz, S., Balakrishnan, N.: Chapter 21: beta distributions. Continuous Univariate Distributions Vol. 2 (1995)
22. Fang, A., Macdonald, C., Ounis, I., Habel, P.: Topics in tweets: A user study of
topic coherence metrics for Twitter data. In: Proc. of ECIR. (2016)
23. Fang, A., Macdonald, C., Ounis, I., Habel, P.: Using word embedding to evaluate
the coherence of topics from twitter data. In: Proc. of SIGIR. (2016)
24. Fang, A., Macdonald, C., Ounis, I., Habel, P.: Examining the coherence of the top
ranked tweet topics. In: Proc. of SIGIR. (2016)
25. AlSumait, L., Barbará, D., Gentle, J., Domeniconi, C.: Topic significance ranking
of LDA generative models. In: Proc. of ECMLPKDD. (2009)