Introduction
Emergence of web services such as blog, microblog and social networking websites allows people to contribute information publicly. This user-generated information is generally more personal, informal and often contains personal opinions. In aggregate, it can be useful for reputation analysis of entities and products, natural disasters detection, obtaining first-hand news, or even demographic analysis. Twitter, an easily accessible source of information, allows users to voice their opinions and thoughts in short text known as tweets.
Latent Dirichlet allocation (LDA) BIBREF0 is a popular form of topic model. Unfortunately, a direct application of LDA on tweets yields poor result as tweets are short and often noisy BIBREF1 , i.e. tweets are unstructured and often contain grammatical and spelling errors, as well as informal words such as user-defined abbreviations due to the 140 characters limit. LDA fails on short tweets since it is heavily dependent on word co-occurrence. Also notable is that text in tweets may contain special tokens known as hashtags; they are used as keywords and allow users to link their tweets with other tweets tagged with the same hashtag. Nevertheless, hashtags are informal since they have no standards. Hashtags can be used as both inline words or categorical labels. Hence instead of being hard labels, hashtags are best treated as special words which can be the themes of the tweets. Tweets are thus challenging for topic models, and ad hoc alternatives are used instead. In other text analysis applications, tweets are often `cleansed' by NLP methods such as lexical normalization BIBREF2 . However, the use of normalization is also criticized BIBREF3 .
In this paper, we propose a novel method for short text modeling by leveraging the auxiliary information that accompanies tweets. This information, complementing word co-occurrence, allows us to model the tweets better, as well as opening the door to more applications, such as user recommendation and hashtag suggestion. Our main contributions include: 1) a fully Bayesian nonparametric model called Twitter-Network (TN) topic model that models tweets very well; and 2) a combination of both the hierarchical Poisson Dirichlet process (HPDP) and the Gaussian process (GP) to jointly model text, hashtags, authors and the followers network. We also develop a flexible framework for arbitrary PDP networks, which allows quick deployment (including inference) of new variants of HPDP topic models. Despite the complexity of the TN topic model, its implementation is made relatively straightforward with the use of the framework.
Background and Related Work
LDA is often extended for different types of data, some notable examples that use auxiliary information are the author-topic model BIBREF4 , the tag-topic model BIBREF5 , and Topic-Link LDA BIBREF6 . However, these models only deal with just one kind of additional information and do not work well with tweets since they are designed for other types of text data. Note that the tag-topic model treats tags as hard labels and uses them to group text documents, which is not appropriate for tweets due to the noisy nature of hashtags. Twitter-LDA BIBREF1 and the behavior-topic model BIBREF7 were designed to explicitly model tweets. Both models are not admixture models since they limit one topic per document. The behavior-topic model analyzes tweets' “posting behavior” of each topic for user recommendation. On the other hand, the biterm topic model BIBREF8 uses only the biterm co-occurrence to model tweets, discarding document level information. Both biterm topic model and Twitter-LDA do not incorporate any auxiliary information. All the above topic models also have a limitation in that the number of topics need to be chosen in advance, which is difficult since this number is not known.
To sidestep the need of choosing the number of topics, BIBREF9 proposed Hierarchical Dirichlet process (HDP) LDA, which utilizes the Dirichlet process (DP) as nonparametric prior. Furthermore, one can replace the DP with the Poisson-Dirichlet process (PDP, also known as the Pitman-Yor process), which models the power-law of word frequencies distributions in natural languages. In natural languages, the distribution of word frequencies exhibits a power-law BIBREF10 . For topic models, replacing the Dirichlet distribution with the PDP can yield great improvement BIBREF11 .
Some recent work models text data with network information ( BIBREF6 , BIBREF12 , BIBREF13 ), however, these models are parametric in nature and can be restrictive. On the contrary, Miller et al. BIBREF14 and Lloyd et al. BIBREF15 model network data directly with nonparametric priors, i.e. with the Indian Buffet process and the Gaussian process respectively, but do not model text.
Model Summary
The TN topic model makes use of the accompanying hashtags, authors, and followers network to model tweets better. The TN topic model is composed of two main components: a HPDP topic model for the text and hashtags, and a GP based random function model for the followers network. The authorship information serves to connect the two together.
We design our HPDP topic model for text as follows. First, generate the global topic distribution $\mu _0$ that serves as a prior. Then generate the respective authors' topic distributions $\nu $ for each author, and a miscellaneous topic distribution $\mu _1$ to capture topics that deviate from the authors' usual topics. Given $\nu $ and $\mu _1$ , we generate the topic distributions for the documents, and words ( $\eta $ , $\theta ^{\prime }$ , $\theta $ ). We also explicitly model the influence of hashtags to words. Hashtag and word generation follows standard LDA and is not discussed here. Note that the tokens of hashtags are shared with the words, i.e. the hashtag #happy share the same token as the word happy. Also note that all distributions on probability vectors are modeled by the PDP, making the model a network of PDP nodes.
The network modeling is connected to the HPDP topic model via the author topic distributions $\nu $ , where we treat $\nu $ as inputs to the GP in the network model. The GP, denoted as $\mathcal {F}$ , determines the links between the authors ( $x$ ). Figure 1 displays the graphical model of TN, where region a and b shows the network model and topic model respectively. See supplementary material for a detailed description. We emphasize that our treatment of the network model is different to that of BIBREF15 . We define a new kernel function based on the cosine similarity in our network model, which provides significant improvement over the original kernel function. Also, we derive a new sampling procedure for inference due to the additive coupling of topic distributions and network connections.
Posterior Inference
We alternatively perform Markov chain Monte Carlo (MCMC) sampling on the topic model and the network model, conditioned on each other. We derive a collapsed Gibbs sampler for the topic model, and a Metropolis-Hastings (MH) algorithm for the network model. We develop a framework to perform collapse Gibbs sampling generally on any Bayesian network of PDPs, built upon the work of BIBREF16 , BIBREF17 , which allows quick prototyping and development of new variants of topic model. We refer the readers to the supplementary materials for the technical details.
Experiments and Applications
We evaluate the TN topic model quantitatively with standard topic model measures such as test-set perplexity, likelihood convergence and clustering measures. Qualitatively, we evaluate the model by visualizing the topic summaries, authors' topic distributions and by performing an automatic labeling task. We compare our model with HDP-LDA, a nonparametric variant of the author-topic model (ATM), and the original random function network model. We also perform ablation studies to show the importance of each component in the model. The results of the comparison and ablation studies are shown in Table 1 . We use two tweets corpus for experiments, first is a subset of Twitter7 dataset BIBREF18 , obtained by querying with certain keywords (e.g. finance, sports, politics). we remove tweets that are not English with langid.py BIBREF19 and filter authors who do not have network information and who authored less than 100 tweets. The corpus consists of 60370 tweets by 94 authors. We then randomly select 90% of the dataset as training documents and use the rest for testing. Second tweets corpus is obtained from BIBREF20 , which contains a total of 781186 tweets. We note that we perform no word normalization to prevent any loss of meaning of the noisy text.
Conclusion and Future Work
We propose a full Bayesian nonparametric Twitter-Network (TN) topic model that jointly models tweets and the associated social network information. Our model employs a nonparametric Bayesian approach by using the PDP and GP, and achieves flexible modeling by performing inference on a network of PDPs. Our experiments with Twitter dataset show that the TN topic model achieves significant improvement compared to existing baselines. Furthermore, our ablation study demonstrates the usefulness of each component of the TN model. Our model also shows interesting applications such as author recommendation, as well as providing additional informative inferences.
We also engineered a framework for rapid topic model development, which is important due to the complexity of the model. While we could have used Adaptor Grammars BIBREF21 , our framework yields more efficient computation for topic models.
Future work includes speeding up the posterior inference algorithm, especially for the network model, as well as incorporating other auxiliary information that is available in social media such as location, hyperlinks and multimedia contents. We also intend to explore other applications that can be addressed with the TN topic model, such as hashtag recommendation. It is also interesting to apply the TN topic model to other types of data such as blog and publication data.
Acknowledgement
We would like to thank the anonymous reviewers for their helpful feedback and comments.
NICTA is funded by the Australian Government through the Department of Communications and the Australian Research Council through the ICT Centre of Excellence Program.

Question: What are the measures of "performance" used in this paper?
Answer: test-set perplexity, likelihood convergence and clustering measures, visualizing the topic summaries, authors' topic distributions and by performing an automatic labeling task