Examining the Coherence of the Top Ranked Tweet Topics
1

Anjie Fang1 , Craig Macdonald2 , Iadh Ounis2 , Philip Habel2

a.fang.1@research.gla.ac.uk, 2 {firstname.secondname}@glasgow.ac.uk
University of Glasgow, UK

ABSTRACT
Topic modelling approaches help scholars to examine thetopics discussed in a corpus. Due to the popularity of Twitter, two distinct methods have been proposed to accommodate the brevity of tweets: the tweet pooling method andTwitter LDA. Both of these methods demonstrate a higher
performance in producing more interpretable topics than thestandard Latent Dirichlet Allocation (LDA) when applied
on tweets. However, while various metrics have been proposed to estimate the coherence of the generated topics fromtweets, the coherence of the top ranked topics, those that are
most likely to be examined by users, has not been investigated. In addition, the effect of the number of generated
topics K on the topic coherence scores has not been studied. In this paper, we conduct large-scale experiments using
three topic modelling approaches over two Twitter datasets,
and apply a state-of-the-art coherence metric to study thecoherence of the top ranked topics and how K affects such
coherence. Inspired by ranking metrics such as precision atn, we use coherence at n to assess the coherence of a topic
model. To verify our results, we conduct a pairwise user
study to obtain human preferences over topics. Our findings are threefold: we find evidence that Twitter LDA outperforms both LDA and the tweet pooling method because
the top ranked topics it generates have more coherence; wedemonstrate that a larger number of topics (K) helps togenerate topics with more coherence; and finally, we showthat coherence at n is more effective when evaluating the coherence of a topic model than the average coherence score.

1.

INTRODUCTION

Topic modelling approaches – e.g. Latent Dirichlet Allocation (LDA) [1] – are widely used to discover latent topics
within a corpus [4, 5]. As Twitter has gained in popularity, scholars have sought out ways to understand and model
discussions on the forum. However tweet corpora are unlike
other corpora (e.g. news articles and books), namely because they are short (limited to 140 characters), and they
contain colloquial phrases and snippet text such as hashtags.
Two well-known topic modelling applications for Twitter
data are Twitter LDA (TLDA) [12] and LDA applied alongside the tweet pooling method (PLDA) [7]. Both methods
have been shown to produce more interpretable topics than
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.

SIGIR ’16, July 17-21, 2016, Pisa, Italyc 2016 ACM. ISBN 978-1-4503-4069-4/16/07. . . $15.00
DOI: http://dx.doi.org/10.1145/2911451.2914731

LDA on Twitter corpora [7, 12]. On the other hand, several
metrics have been proposed in the literature to automatically estimate the coherence of the generated topic models. For example, Newman et al. [8] proposed a Pointwise
Mutual Information(PMI)-based metric using Wikipedia asa background dataset to evaluate the coherence of a topic
from news articles and books. More recently, a new coherence PMI-based metric using a Twitter background has been
proposed for tweet corpora, and was found to be the closest
to human judgements [2].
However, the coherence of the top ranked topics fromtweets, those most likely to be examined by users, has notbeen previously investigated, nor has the effect of the number of generated topics K on the topic coherence scores. In
this paper, we conduct large-scale experiments on two Twitter datasets to investigate the coherence of ranked topics
generated by three topic modelling approaches (LDA, TLDA
and PLDA). Inspired by the precision at n evaluation metric, we also explore the coherence at n scores of the generated topic models by using the state-of-the-art Twitter PMIbased coherence metric [2], which we describe in Section 3.
The contributions of this paper are as follows: 1) we examine which of the three existing topic modelling approaches
for Twitter data generates more coherent topics, 2) we analyse the relationship between the coherence of a topic modeland the number of topics (K), and 3) we evaluate the utility of the coherence at n coherence metric for a topic model.
To validate our findings, we perform a pairwise preferenceuser study conducted on a crowdsourcing platform, where
the workers are asked to choose the more coherent topic in a
topic pair. The obtained human judgements are then compared with the coherence score of the generated topic models. We show, first, that TLDA performs better than LDA
and PLDA; second, that a higher topic number K helps to
produce topics with higher coherence; and finally, that thecoherence at n scores are more effective than the commonlyused averages in evaluating the coherence of a topic model.

2.

BACKGROUND & RELATED WORKLDA [1] models latent topics across terms and documents,
where a topic is described as a probability distribution overterms Φ, and a document has its own probability distribution over topics. LDA has been used extensively to extract
latent topics in standard corpora such as news articles andbooks [5]. Because tweets can be of lower quality due to thebrevity of their content [6, 12], topics generated by using
LDA are likely to be both mixed [11] and harder to interpret by the associated terms. In order to improve the performance of LDA for Twitter data, Mehrotra et al. [7] haveproposed the tweet pooling method, which groups tweets into
virtual documents (e.g. sharing the same author or the same
hashtag). We refer to the application of LDA alongside thetweet pooling method as PLDA. TLDA is another approachfrom Zhao et al. [12], where a background term distribution
is used to distinguish “real” topic terms from backgroundterms, with the assumption that a single tweet contains asingle topic. Indeed both PLDA and TLDA have outperformed LDA in producing more coherent topics [7, 12].
Newman et al. [8] conducted a user study to evaluate several proposed coherence metrics based on capturing the semantic similarity of the topics using external sources, e.g.
Wikipedia and WordNet. They found that the most suitable coherence metric was based on estimating the Pointwise Mutual Information (PMI) of word pairs in a topic for
corpora such as news articles. Stevens et al. [9] used thesame Wikipedia PMI-based metric to analyse the coherenceof the generated topic models from news articles.
Recently, a large-scale user study in [2] evaluated several
coherence metrics including the Wikipedia PMI-based and
WordNet-based metrics on tweets. Fang et al. [2] showedthat a newly proposed coherence metric leveraging a Twitter background dataset, called the Twitter PMI-based metric (hereafter, T-PMI), has a markablely high agreementwith human judgements on tweet corpora. In the following,
we use the T-PMI metric to evaluate the coherence of the
topics generated by using three topic modelling approaches:
LDA, TLDA, and PLDA. We further study the coherenceof the top ranked topics likely to be examined by users, andthe effect of the number of generated topics K on the topic
coherence scores. To the best of our knowledge, this paper
contributes the first study of the coherence of the top ranked
topics and how K affects such coherence on tweets.

3.

METRICS

We now describe the T-PMI coherence metric, which analyses the coherence of topics and topic models. In the TPMI metric [2], a topic t is first represented by the 10 mostfrequent words {w1 , w2 , ..., w10 } selected by the topic termprobabilities in Φ. Any two words among the 10 words ina topic make a word pair, e.g. Pair(wi ,wj ). The coherence
C of a topic is measured by averaging the PMI score of all
its word pairs using Equation (1), where we use the top 10
words representing a topic, i.e. n = 10. The PMI score of aword pair is pre-calculated using Equation (2) from a Twitter background dataset. We describe how we set up thismetric in Section 4.
Pn Pn
1
C(t) = Pn−1(1)
i=1
j=i+1 P M I(wi , wj )m
m=1p(w ,w )ij
P M I(wi , wj ) = log p(wi )×p(wj)(2)

We use the three aforementioned topic modelling approaches to obtain the topic models from tweets, varying the topic
number K. We use the T-PMI coherence metric to calculate
the average coherence of a topic model, i.e. the coherence
score across all K topics. Intuitively, the average coherence of a topic model reflects the quality of the entire topic
model (containing K topics). However, end-users typicallyare only interested in the most coherent topics in a topic
model, rather than the whole model. Inspired by ranking
metrics such as the precision at n metric, we use coherence
at n to evaluate the coherence of a topic model. In particular, coherence at n (coherence@n) indicates the averagecoherence score of the top n most coherent topics, where a
topic is ranked by its coherence score. We argue that coherence at n can more effectively capture the coherence of a
Twitter topic model.

Table 1: Two used Twitter datasets.
Name

Time Period

Users#

Tweets#(1) MAY(2) TVD

1 to 31 May 2015
8 to 10pm, 02 Apr 2015

2,452
121,594

334,922
343,511

4.

DATASETS & EXPERIMENTAL SETUP

We use two datasets in our experiments. The first1 iscomprised of the tweets of 2,452 newspaper journalists in
New York posted from 01/05/2015 to 31/05/2015, denoted
here as MAY. We can reasonably assume that journalists
discussed a large number of topics over this one-month period. The second dataset1 consists of tweets related to thefirst TV debate among political party leaders during the UK
General Election of 2015, denoted here as TVD. We expectthat the number of topics covered over this short time period to be more limited, as conversations on Twitter tended
to focus around the issues introduced during the two-hourdebate. Details of these two datasets are shown in Table 1.
We use a Twitter background dataset from [2, 3], whichcontains 1%-5% random tweets posted from 01/01/ 2015 to
30/06/2015. Following [2, 3], we remove stopwords, terms
occurring in less than 20 tweets and the retweets from thisbackground dataset. The remaining tweets (30,151,847) areused to calculate the prior PMI score of the occurring wordpairs in order to implement the T-PMI metric.
We use the Mallet2 and Twitter LDA3 toolkits to apply
the three topic modelling approaches. For PLDA, we groupthe tweets posted by the same user in a given time interval
into a virtual document4 . The time interval is set to 10 minutes for TVD, and 6 hours for MAY, given the narrow time
period of the TVD dataset and the more expansive one for
the MAY dataset. The LDA parameters α and β are set to
50/K and 0.01 following [10], and the TLDA parameter γ isset to 20 following [12]. Since the TVD dataset contains just
two hours of tweets, we set the maximum topic number K
to 100, and then use 46 different K values between 10 and
100 (step = 2). We set K to a maximum of 500 for MAY,
and use 49 different K values ranging from 10 to 500 (step= 10). Each topic modelling approach is run 5 times foreach K. Thus, we obtain 5 topic models for each K. In thenext section, we analyse the coherence of these 1,425 topic
models (46 × 5 × 3 + 49 × 5 × 3).

5.

COHERENCE OF TOPIC MODELS

Figure 1 shows the T-PMI coherence of three types of
topic models (LDA, TLDA, and PLDA) for MAY and TVD,
varying K. Each point in Figure 1 represents the average coherence or coherence@n score of 5 topics models, forn = {5, 10, 20, 30, 40}. For instance, the point (20, 0.0020)
on the red line in Figure 1(a) shows that the average coherence score of the top 5 topics (coherence@5) in the 5 topic
models where K=20 is 0.0020. Higher scores indicate better
coherence.
First, it is clear that the average coherence (the solid blackline) of all topics in a model decreases as K increases acrossthe three topic modelling approaches. These results are similar to what Stevens et al. [9] observed. However, the coherence@n score (represented by coloured lines with distinguishing symbols) of all topic models increases as K grows.
This finding tells us that setting a higher number of topics,
1

Collected using the Twitter API. 2 mallet.cs.umass.edu
github.com/minghui/Twitter-LDA 4 We do not group
tweets by hashtags since there are not many hashtags in ourdatasets.
3

Average
Coherence

Coherence@5

Coherence@10

Coherence(a). LDA models on the TVD dataset

0.00300.0030Coherence@20

Coherence@30
Coherence(b). TLDA models on the TVD dataset

0.00300.00250.00250.00250.00200.00200.00200.00150.00150.00150.00100.00100.00100.000510
Coherence

20

30

40

50

60

70

80

90

100
K

(d). LDA models on the MAY dataset

0.008

0.000510
Coherence

0.008

20

30

40

50

60

70

80

90

100
K

0.000510
Coherence(e). TLDA models on the MAY dataset

0.008

0.007

0.007

0.007

0.006

0.006

0.006

0.005

0.005

0.005

0.004

0.004

0.004

0.003

0.003

0.003

0.002

0.002

0.002

0.001

0.001

10

100

200

300

400

500

20

30

40

50

60

70

80

90

100
K

(f). PLDA models on the MAY dataset

0.001

10

K

Coherence@40(c). PLDA models on the TVD dataset

100

200

300

400

500
K

10

100

200

300

400

500
K

Figure 1: The coherence of three types of topic models with varying K over two datasets.
k=10
%

0.04

0.04

50
40
30
20
10
0
0.00

k=60

k=80

0.08

0.11

0.15

0.19

0.23

0.27

0.30

0.08

0.11

0.15

0.19

0.23

0.27

0.30

(c). PLDA models on the TVD dataset

%

50
40
30
20
10
0
0.00

k=40

(b). TLDA models on the TVD dataset

%

50
40
30
20
10
0
0.00

k=20

k=100

(a). LDA models on the TVD dataset

0.04

0.08

0.11

0.15

0.19

0.23

0.27

0.30

0.34
0.38
Coherence, 10 ¡2

0.34
0.38
Coherence, 10 ¡2

ics with less coherence. This is why the average coherence ofthe topic models declines as K increases, shown in Figure 1
by the solid black lines. Second, the volume of topics withcoherence [0, 0.4−3 ) is lowest in the TLDA models. This
result further indicates that TLDA outperforms PLDA and
LDA, as it generates fewer meaningless topics. We also observe the same pattern for the MAY dataset5 .
To verify that a larger K helps generate topics with higher
coherence, and to examine the utility of the coherence@nmetric, we conduct a user study in Section 6 where we compare human preferences with the topics’ coherence scoresfrom the T-PMI metric.

6.

0.34
0.38
Coherence, 10 ¡2

Figure 2: The coherence values distribution.
K, increases the coherence of the n most coherent topics.
Second, the coherence@n score is higher for TLDA acrossthe two datasets, which suggests that the top n topics in the
TLDA models have a higher coherence. We also see that
PLDA generates slightly more coherent topics than LDA,
although the performance of PLDA is closer to LDA than
to TLDA. Third, we observe that the average coherence andcoherence@n score of the LDA & PLDA topic models on the
TVD dataset become stable around K=80, while the coherence of the TLDA topic models on the MAY dataset has a local peak around K=390. It should be noted that the averagecoherence score cannot adequately capture the performancedifferences of the 3 topic modelling approaches on the MAY
dataset (see Figure 1(d), (e) and (f)), and so coherence@n ispreferred. Since a larger K leads to a higher computationalcost, a K should be selected when the coherence of the topic
model begins to stabilise or when it reaches a local peak.
Next, in Figure 2, we show the distribution of the topics’ coherence scores for topic models with varying K on the
TVD dataset. The coherence scores are distributed across
10 bins, to show the number of topics exhibiting different
levels of coherence. First, the volume of topics with coherence [0, 0.4−3 ) is highest across all topic models in the TVD
dataset. As K increases, the topic models include more top-

USER STUDY

Because it is challenging for humans to give a graded coherence score for topics, we conduct a pairwise preferenceuser study. Similar studies have been conducted previously
in [2, 3]. We recruited workers from CrowdFlower6 andasked them to select the more coherent topic from two provided topics (a topic pair). We describe here how we generate the topic pairs, the CrowdFlower job, how we control
the job quality, and the crowdsourcing results.
Generating Topic Pairs For a given approach (e.g.
TLDA), we choose two K = {a,b} (a < b) values representing different topic model outcomes with difference coherences (recall that each approach is repeated 5 times, hence
10 models in total). From each selected topic model, we
select the top n most coherent topics. Thus, we have two
topic pools: Pk=a & Pk=b , and each pool has 5 × n topics7 . To make the preference task easier for workers, weshow two similar topics in a topic pair. First, we sample
a number of topics from Pk=a randomly. For each samP
pled topic (tj k=a ), we use Equation (3) to identify its closest topic in Pk=b , where Vt is a vector representation using
the term distribution of topic t. We denote the selectedpairs of topics as Pairs(Pk=a → Pk=b ). Likewise, we also
generate the same number of Pairs(Pk=b → Pk=a ). In ouruser study, we use TLDA topic models on the MAY datasetwith K={50,100,300,390}, since its K range is set wider
than that of the TVD dataset. We compare the coherenceof models with K=50 vs. K=300 (denoted as comparison
5
6

The figure is not shown due to space limitations.
crowdflower.com 7 Each experiment is repeated 5 times.

Table 2: The comparison of coherence scores.

Figure 3: The CrowdFlower user interface.
Unit(50,300)) and topic models with K=100 vs. K=390(comparison Unit(100,390)). Therefore, we can examine
whether topic models with a larger K (300/390) have more
coherent topics than the models with smaller K (50/100).
For the comparisons Unit(50,300)/Unit(100,390), we selectthe top 30/208 topics for the topic pools (Pk={50,100,300,390} ).
We generate 40 topic pairs for each comparison unit. Notethat we ignore the topic pairs where two topics in the pairshare the same top 10 words or share less than 3 mutual
words among the top 10 words. Hence this ensures that thetwo selected topics in a topic pair are similar enough forhumans.
Pclosest(tj k=a ) = argmaxi<K (cosine(VtPk=a , VtPk=b ))
j

i

(3)

Job Description We present each CrowdFlower worker
with the top 10 words (ranked by their probabilities in a
topic) from two topics in a topic pair, labelled Topic 1 andTopic 2, along with their 3 most retweeted tweets9 . Basedon these 10 words, we ask the workers to select the more
coherent topic among the two shown. We tell the worker
that a more coherent topic is one that is less mixed and thatcan be interpreted. The workers are instructed to take into
account: 1) the number of semantically similar words (e.g.
President & Obama) among the 10 shown words, 2) whether
the words shown suggest a mixed topic (i.e. more than one
discussion), and 3) whether the words shown are more specific. A worker can also consider two associated tweets forthe two topics if he/she cannot make a decision. We also offer guidance for using these tweets: 1) whether the 10 shownwords are reflected by their tweets and 2) whether thesetweets are related with the two topics. Figure 3 captures
the user interface on CrowdFlower, along with an example
of a topic pair. We collect a total of 5 judgements from 5
different workers for each topic pair. For each judgement,
we paid workers $0.05.
Quality Control We used test questions for worker quality control. To set these test questions, we began by choosing
a number of topic pairs, where the topic preference was manually verified in advance. Only those workers who passed thetest were allowed to enter the task. The worker must havemaintained more than 70% accuracy on the test questions
throughout the whole task, otherwise their judgements were
not used. We limited the workers’ country to the UnitedStates as the MAY tweets were written by New York journalists. Our user study engaged 52 trusted workers.
Crowdsourcing Results Table 2 lists the human judgement results compared with the coherence scores from the
T-PMI metric. For comparison unit (50,300) - Table 2(a)- the 40 topics we select from topic models with K=300
are significantly more coherent than those from topic models with K=50 according to both the human vote10 fraction
8

The top 20 topics in Unit(100,390) are more distinguishable than the top 30. 9 These tweets help workers understand the topic. 10 A topic in a topic pair receives one votewhen it is preferred by a human.

(a). TLDA topic models with K=50 vs. K=300, Unit(50,300)
K
Human vote fraction
T-PMI
50
0.311
2.06−3
(∗)
(∗)
300
0.689
3.15−3(b). TLDA topic models with K=100 vs. K=390, Unit(100,390)
K
Human vote fraction
T-PMI
100
0.411
3.54−3
(∗∗)
(∗∗)
390
0.589
4.00−3∗/(∗∗) denote p < 0.01 (p < 0.05) according to the Wilcoxon
signed-rank test, compared to the smaller K.

and the T-PMI coherence scores. We observe the same results for comparison unit (100,390), c.f. Table 2(b). Thisfinding shows again that a larger K helps to obtain topics
with more coherence. However, if one uses only the average coherence score to evaluate the topic model, then thesemodels are indistinguishable. This finding suggests that thecoherence@n metric should be used to measure the coherence quality of a topic model.

7.

CONCLUSIONSThis paper studied the coherence of Twitter topic models through a large scale experiment varying both the topic
modelling approach and K, and thereafter verified conclusions using a pairwise user study. To summarise, we firstfound that Twitter LDA (TLDA) outperformed PLDA and
LDA, as it generated topics with a higher coherence and alsoless meaningless topics. Second, we showed that increasing
the number of topics (K) helped to generate topics witha higher coherence. Third, coherence at n is more effective in evaluating a topic model than the average coherence.
Our paper has two implications for researchers implementing topic modelling on tweets: first, larger K values result inthe top n topics being the most coherent (n < K); second,
when evaluating the coherence of a topic model, scholars
should use the coherence at n metric.

8.

REFERENCES[1] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet
allocation. the Journal of machine Learning research,
3:993–1022, 2003.
[2] A. Fang, C. Macdonald, I. Ounis, and P. Habel. Topics intweets: A user study of topic coherence metrics for Twitter
data. In Proc. of ECIR, 2016.
[3] A. Fang, C. Macdonald, I. Ounis, and P. Habel. Using wordembedding to evaluate the coherence of topics from Twitter
data. In Proc. of SIGIR, 2016.
[4] A. Fang, I. Ounis, P. Habel, C. Macdonald, and
N. Limsopatham. Topic-centric classification of Twitter user’s
political orientation. In Proc. of SIGIR, 2015.
[5] T. L. Griffiths and M. Steyvers. Finding scientific topics. In
Proc. of NAS, 2004.
[6] L. Hong and B. D. Davison. Empirical study of topic modelingin Twitter. In Proc. of SOMA, 2010.
[7] R. Mehrotra, S. Sanner, W. Buntine, and L. Xie. Improving
LDA topic models for microblogs via tweet pooling andautomatic labeling. In Proc. of SIGIR, 2013.
[8] D. Newman, J. H. Lau, K. Grieser, and T. Baldwin. Automatic
evaluation of topic coherence. In Proc. of NAACL, 2010.
[9] K. Stevens, P. Kegelmeyer, D. Andrzejewski, and D. Buttler.
Exploring topic coherence over many models and many topics.
In Proc. of EMNLP-CoNLL, 2012.
[10] M. Steyvers and T. Griffiths. Probabilistic topic models.
Handbook of latent semantic analysis, 427:424–440, 2007.
[11] J. Tang, Z. Meng, X. Nguyen, Q. Mei, and M. Zhang.
Understanding the limiting factors of topic modeling via
posterior contraction analysis. In Proc. of ICML, 2014.
[12] W. X. Zhao, J. Jiang, J. Weng, J. He, E.-P. Lim, H. Yan, and
X. Li. Comparing Twitter and traditional media using topic
models. In Proc. of ECIR, 2011.