Venue: ACL 2020

# An Online Semantic-enhanced Dirichlet Model for Short Text Stream Clustering

Jay Kumar and Junming Shao\({}^{*}\) and Salah ud din

Data Mining Lab, School of Computer Science and Engineering,

University of Electronic Science and Technology of China, Chengdu, 611731, China

jay_tharwani1992@yahoo.com, junmshao@uestc.edu.cn

Wazir Ali

SMILE Lab, School of Computer Science and Engineering,

University of Electronic Science and Technology of China, Chengdu, 611731, China

###### Abstract

Clustering short text streams is a challenging task due to its unique properties: infinite length, sparse data representation and cluster evolution. Existing approaches often exploit short text streams in a batch way. However, determine the optimal batch size is usually a difficult task since we have no prior knowledge when the topics evolve. In addition, traditional independent word representation in the graphical model tends to cause "term ambiguity" problem in short text clustering. Therefore, in this paper, we propose an Online Semantic-enhanced Dirichlet Model for short text stream clustering, called OSDM, which integrates the word-occurrence semantic information (i.e., context) into a new graphical model and clusters for each arriving short text automatically in an online way. Extensive results have demonstrated that OSDM gives better performance compared to many state-of-the-art algorithms on both synthetic and real-world data sets.

+
Footnote †: Corresponding author: Junming Shao

_Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 766–776

July 5 - 10, 2020. ©2020 Association for Computational Linguistics

## 1 Introduction

A massive amount of short text data is constantly generated with online social platforms such as microblogs, Twitter and Facebook. Clustering of such short text streams has thus gained increasing attention in recent years due to many real-world applications like event tracking, hot topic detection, and news recommendation Hadifar et al. (2019). However, due to the unique properties of short text streams such as infinite length, evolving patterns and sparse data representation, short text stream clustering is still a big challenge Aggarwal et al. (2003); Mahdiraji (2009).

During the past decade, many approaches have been proposed to address the text stream clustering problem from different points of view, and each method comes with specific advantages and drawbacks. Initially, traditional clustering algorithms for static data were enhanced and transformed for text streams Zhong (2005). Very soon, they are replaced by model-based algorithms such as LDA Blei et al. (2003), DTM Beli and Lafferty (2006), TDPM Ahmed and Xing (2008), GSDMM Yin and Wang (2016), DPMPF Huang et al. (2013), TM-LDA Wang et al. (2012), NPMM Chen et al. (2019) and MStream Yin et al. (2018), to mention a few. However, for most established approaches, they often work in a batch way, and assume the instances within a batch are interchangeable. This assumption usually cannot hold for topic-evolving text data corpus. Determining an optimal batch size is also a non-trivial task for different text streams Howard and Ruder (2018).

Additionally, unlike long text documents, short text clustering further suffers from the lack of supportive term occurrence to capture semantics Gong et al. (2018). For most existing short text clustering algorithms like Sumblr Shou et al. (2013), DCT Liang et al. (2016) and MSreamF Yin et al. (2018), exploiting independent word representation in their cluster models tends to cause ambiguity. Let us show the following four tweets, for example:

T1: _"A regular intake of an Apple can improve your health and muscle stamina."_ T1: _"A glass of fresh apple juice is recommended for breakfast."_ T2: _"New Apple Watch can monitor your health."_

* _"Apple will launch new smartphone iPhoneX this december."_

Tweets of these two topics share few common terms, i.e., '_health_' or '_apple_'. It creates an ambiguity if the model deals with only single term representation to calculate the similarity. However, the co-occurring terms representation (i.e., context) helps a model to identify the topic1 correctly.

Footnote 1: Topic and cluster will be interchangeably used in this paper

To solve these aforementioned issues, we propose an online semantic-enhanced dirichlet model for short text stream clustering. Compared to existing approaches, it has following advantages. (1) It allows processing each arriving short text in an online way. The online model is not only free of determining the optimal batch size, but also lends itself to handling large-scale data streams efficiently; (2) To the best of our knowledge, it is the first work to integrate semantic information for model-based online clustering, which is able to handle "term ambiguity" problem effectively and finally support high-quality clustering; (3) Equipped with Poly Urn Scheme, the number of clusters (topics) are determined automatically in our cluster model.

## 2 Related Work

During the past decade, many text stream clustering algorithms have been proposed. Here, due to the space limitation, we only report some model-based approaches which are highly related to our work. For more details, please refer to comprehensive surveys, e.g., Mahdiraji (2009); Silva et al. (2013); Nguyen et al. (2015); Aggarwal (2018).

The early classical attempt for text clustering is Latent Dirichlet Allocation (LDA) Blei et al. (2003). However, it cannot handle the temporal data for text streams. For this purpose, many LDA variants have been proposed to consider the text streams such as dynamic topic model (DTM) Blei and Lafferty (2006), dynamic mixture model (DMM) Wei et al. (2007), temporal LDA (T-LDA) Wang et al. (2012), streaming LDA (S-LDA) Amoualian et al. (2016), and dirichlet mixture model with feature partition (DPMPF) Zhao et al. (2016). These models assume that each document contains rich content, and thus they are not suitable for dealing with the short text streams. Later, Dirichlet multinomial mixture model-based dynamic clustering topic (DCT) model was designed to deal with short text streams by assigning each document with single topic Liang et al. (2016). Very soon, GSDMM was proposed to extend DMM with collapsed gibbs sampling to infer the number of clusters Yin and Wang (2014). However, most of these models did not investigate the evolving topics (clusters) in text streams where the number of topics usually evolves over time.

To automatically detecting the number of clusters, Ahmed and Xing (2008) proposed a temporal dirichlet process mixture model (TDMP). It divides the text stream into many chunks (batches), and assumes that the documents inside each batch are interchangeable. Later, GSDPMM was proposed with collapsed gibbs sampling to infer the number of clusters in each batch. In contrast to LDA, GSDPMM not only converges faster but also dynamically assigns the number of clusters over time Yin and Wang (2016a). However, both TDMP and GSDPMM models do not examine the evolving topics, and, these models process the text stream for multiple times. Thereafter, MStreamF Yin et al. (2018) was thus proposed by incorporating a forgetting mechanism to cope with cluster evolution, and allows processing each batch only one time. The NPMM model Chen et al. (2019) was recently introduced by using the word-embeddings to eliminate a cluster generating parameter of the model.

In summary, for most existing approaches, they usually work in a batch way. However, determining optimal batch sizes for different text streams is usually a difficult task. More importantly, due to the intrinsic sparse data representation of short-text data, the semantics, is little investigated in established approaches. Actually, they need to be carefully considered to decrease the term ambiguity in short text clustering.

## 3 Preliminaries

Here, the problem statement is first given, followed with a brief introduction about dirichlet process and Poly Urn scheme.

### Problem Formulation

Formally, a text stream is continuous arrival of text documents over time: \(S_{t}=\left\{d_{t}\right\}^{\infty}_{t=1}\). Where \(d_{t}\) denotes a document arrived at time \(t\). Each document contains specific words \(d_{t}=\left\{w_{1},w_{2},\ldots,w_{n}\right\}\) and may have different length. The key objective of the clustering task is to group similar documents into clusters: \(Z=\left\{z_{t}\right\}^{\infty}_{t=1}\), and each clus 

ter \(z_{t}\) contains documents represented as \(z_{t}=\left\{d^{z_{t}}_{1},d^{z_{t}}_{2},\ldots,d^{z_{t}}_{n}\right\}\). For short text clustering, each document is the member of only one topic, so \(z_{i}\cap z_{j}=\phi\), where \(i\neq j\).

### Dirichlet Process

Dirichlet Process (DP) is a non-parametric stochastic processes to model the data Teh et al. (2006). It is the process to draw a sample from (base) distribution, where each sample itself is a distribution, denoted as \(\mathcal{N}\sim\mathrm{DP}(\alpha,\mathcal{N}_{0})\). Here, \(\mathcal{N}\) is the drawn sample from the base distribution \(\mathcal{N}_{0}\). The drawing procedure of a sample from the distribution is controlled by a concentration parameter \(\alpha\).

### Poly Urn Scheme (PUS)

The procedure to draw the sequential samples \(\mathcal{N}_{1},\mathcal{N}_{2}\dots\) from a distribution is described by the _poly unr scheme_Blackwell et al. (1973). It can be summarized as:

Here, \(\delta(x)=1\) if \(x=0\) and \(\delta(x)=0\) otherwise. Initially, the un is empty, so we draw a color from the base distribution i.e. \(\mathcal{N}_{1}\sim\mathcal{N}_{0}\), and put a ball of drawn color into the unrn. In the next turn, either we draw a color from the distribution which is already drawn with probability of \(\frac{n-1}{n+n-1}\) , or draw a new color with probability of \(\frac{\alpha\mathcal{N}_{0}}{\alpha+n-1}\). Since, drawing samples from distribution is repeated, so the same color may appear more than once. This defines that we have \(K\) number of distinct colors and \(n\) number of draws. This condition is defined by a well-known process called Chinese restaurant process (CRP) Ferguson and Thomas S Ferguson (1973). In CRP, we suppose that there are infinite number of tables in a restaurant, and each table surrounds infinite number of empty chairs. The first customer sits on first table, and later on the next customer either chooses to sit on any occupied table with probability of \(\frac{n_{k}}{\alpha+n-1}\) or chooses an empty table with probability of \(\frac{\alpha}{\alpha+n-1}\). Here, \(n_{k}\) is number of customers sitting on a specific table. A new customer is tend to be attracted towards a highly crowded table. This phenomenon is one part of our equation to understand creation of clusters over time. The CRP represents the draws from distribution \(\mathcal{G}\), while the stick-breaking process shows the property of \(\mathcal{G}\) explicitly:

\[\mathcal{G}(\mathcal{N})=\sum_{k=1}^{\infty}\theta_{k}\delta\left(\mathcal{ N}-\mathcal{N}_{k}\right),\quad\mathcal{N}_{k}\sim\mathcal{N}_{0} \tag{1}\]

The mixture weights \(\theta=\left\{\theta_{k}\right\}_{k=1}^{\infty}\) can be formalized by \(\theta\sim GEM(\gamma)\)Neal (2000). We exploit Equation (1) for the generative process of the Dirichlet process multinomial mixture model (DPMM) as follows.

\[z_{d}|\theta\sim\operatorname{Mult}(\theta) d=1,\ldots,\infty\] \[\mathcal{N}_{k}|\beta\sim\text{Dir}(\beta) k=1,\ldots,\infty\] \[d\left|z_{d},\left\{\mathcal{N}_{k}\right\}_{k=1}^{\infty}\sim p \left(d|\mathcal{N}_{z_{d}}\right)\right.\]

Here, \(z_{d}\) is the assigned documents to the cluster, which are multinomial distributed. The probability of document \(d\) generated by topic \(z\) is summarized as:

\[p\left(d|\mathcal{N}_{z}\right)=\prod_{w\in d}\operatorname{Mult}\left(w| \mathcal{N}_{z}\right) \tag{2}\]

Here, the naive Bayes assumption is considered where words in a document are independently generated by the topic. Whereas, the sequential draw of the sample can be derived by following the CRP. It is also assumed that the position of words in a document is not considered while calculating the probability.

## 4 Proposed Approach

This section gives a brief discussion about the representation and formulation of the proposed algorithm.

### Model Representation

We build our model upon the DPMM Yin and Wang (2016a), which is an extension of the DMM model to deal with evolving clusters. We call our model as OSDM (Online Semantic-enhanced Dirichlet Model), aiming at incorporating the semantic information and cluster evolution simultaneously for short text stream clustering in _an online way_. The graphical model of OSDM is given in Figure 0(a).

We show two major differences in our model to highlight the novelty. First, for word-topic distribution, we embed semantic information by capturing the ratio of word co-occurrence. Thereby, independent word generating process and word co-occurrence weight are well considered in topic generation. Secondly, 

by instance fashion to cluster the documents, instead of batch by batch. For comparison, Figure 1b further show the MStreamF Yin et al. (2018) model. At initial stage before clustering documents of a batch, MStreamF update vocabulary set (active terms) from all the documents in a batch, then it starts the clustering each document of the batch. However, OSDM does not consider fixed number of documents to create vocabulary set, instead it incrementally updates with each arriving document.

### Model Formulation

Defining the relationship between documents and clusters is the most crucial task while dealing with the text stream clustering problem. The threshold-based methodology Nguyen et al. (2015) adapts similarity measures to define the homogeneity threshold between a cluster and a document. If the dissimilarity between the exiting clusters and a new arriving document is above the threshold, then a new cluster is created. However, due to the dynamic nature of the stream, it is very hard to define the similarity threshold manually.

In contrast, we assume that documents are generated by DPMM (see Section 3). Most recent algorithm MSTreamF improved DPMM to cluster short text documents in the stream. As a further study, we integrate the semantic component in DPMM model. Additionally, we integrate term importance on the basis of cluster frequency. The derived equation for calculating the probability of a document \(d\) choosing existing cluster \(z\) is given in Equation (3).

\[p\left({z_{d}=z|\vec{z},\vec{d},\alpha,\beta}\right)=\left({m_{ z}\over D-1+\alpha D}\right)\cdot\\ \left({\prod_{w\in d}\prod_{j=1}^{N_{d}^{w}}{(n_{z}^{w}\cdot ICF_ {w})+\beta+j-1\over\prod_{i=1}^{N_{d}}{n_{z}}+V\beta+i-1}}\right)\cdot\\ \left({1+\sum_{w_{i}\in d\wedge w_{j}\in d}cw_{ij}}\right) \tag{3}\]

The first term of this Equation \(\left({m_{z}\over D-1+\alpha D}\right)\) represents completeness of the cluster. Here, \(m_{z}\) is the number of documents contained by the cluster \(z\) and \(D\) is the number of current documents in active clusters2. Whereas, \(\alpha\) is the concentration parameter of the model. The middle term of the equation based on multinomial distribution (see Equation (2)) with psuedo weight of words \(\beta\) defines the homogeneity between a cluster and a document. \(N_{d}\) and \(N_{d}^{w}\) represents total number of words and term frequency of word \(w\) in document \(d\), respectively. The symbol \(n_{z}^{w}\) is the term frequency of the word \(w\) in the cluster \(z\). The current vocabulary size of the model is represented by \(V\). \(n_{z}\) is the number of words in the cluster \(z\). \(ICF_{w}\) calculates the term importance over the active clusters in the model, which is defined as follows.

Footnote 2: Active clusters refer to those clusters which are not yet deleted from the model.

\[ICF(w\in d)=\log\left({|Z|\over|weZ|}\right) \tag{4}\]

Here, \(|Z|\) represents the number of active clusters in the model. The denominator part of Equation (4) is the number of those cluster which contains the word \(w\). The term \(\left({1+\sum_{w_{i}\in d\wedge w_{j}\in d}cw_{ij}}\right)\) defines the semantic weight of term co-occurrence between the cluster and a document. Formally, we define a value of an entry \(cw_{ij}\) in the co-occurrence matrix as follows.

\[cw_{ij}={\sum\limits_{d^{\prime}\subseteq z}{n_{d^{\prime}}^{w_{ i}}}\over\sum\limits_{d^{\prime}\subseteq z}{n_{d^{\prime}}^{w_{i}}}+\sum \limits_{d^{\prime}\subseteq z}{n_{d^{\prime}}^{w_{j}}}}\quad(w_{i},w_{j})\in d ^{\prime} \tag{5}\]

Here, \(n_{z}^{d^{\prime}}\) is frequency count of word \(w_{i}\) in document \(d^{\prime}\). The ratio between \(w_{i}\) and \(w_{j}\) must satisfy the property \(cw_{ij}+cw_{ji}=1\) . We calculate the term co-occurrence weight of those terms which are

Figure 1: The graphical representation of OSDM and MStream. Here MStream works in a batch way while OSDM works in an online way.

 

common in the cluster \(z\) and document \(d\). Term co-occurrence matrix is constructed where two terms are co-occurred in a single document. Therefore, if the size of cluster feature set (discussed in Section 4.3) is \(|V_{z}|\), then it is not necessary that the co-occurrence matrix would be \(|V_{z}|\times|V_{z}|\).

So far, we have defined the probability of a document choosing existing cluster, then we have to define the probability for a document to creating a new cluster. By following the DPMM for infinite number of clusters, which transform \(\theta\sim GEM(\gamma)\) into \(\theta\sim GEM(\alpha D)\), because the hyper-parameter for the mixture model should be dynamically change over time. Therefore, the probability of creating a new cluster is as follows.

\[p\left(z_{d}=z|\vec{z}_{\neg d},\vec{d},\alpha,\beta\right)= \left(\frac{\alpha D}{D-1+\alpha D}\right)\\ \cdot\left(\frac{\prod_{w\in d}\prod_{j=1}^{N^{w}_{d}}\beta+j-1}{ \prod_{i=1}^{N_{d}}V\beta+i-1}\right) \tag{6}\]

Here, the pseudo number of clusters related documents in the model is represented as \(\alpha D\) , and \(\beta\) is the pseudo term frequency of each word (exist in document) of the new cluster.

### The cluster feature (CF) set

The similarity-based text clustering approaches usually follow vector space model (VSM) to represent the cluster feature space Din and Shao (2020). However, a topic needs to be represented as the subspace of global feature space. Here, we use a micro-cluster feature set to represent each cluster. Namely, a cluster is represented as the summary statistics of a set of words of related documents. In our model, a cluster feature (CF) set is defined as a 6-tuple \(\{m_{z},n_{z}^{w},cw_{z},len_{z},l_{z},u_{z}\}\), where \(m_{z}\) is the number of documents in the cluster \(z\), \(n_{Z}^{w}\) is the number of frequency of the word \(w\) in the cluster, \(cw_{z}\) is the word to word co-occurrence matrix, \(len_{z}\) is the number of words in the cluster \(z\) which is sum of all frequencies of words, \(l_{z}\) is the cluster weight, and \(u_{z}\) is the last updated time stamp.

```
Input:\(S_{t}:\{d_{t}\}_{t=1}^{\infty}\) , \(\alpha:\) concentration parameter, \(\beta:\) pseudo weight of term in cluster, \(\lambda:\) decay factor Output: Cluster assignments \(z_{d}\)
1\(K=\phi\)while\(d_{t}\) in \(S_{t}\)do
2\(t=t+1\)
3\(K=removeOldZ_{i}(K)\)
4\(K=reduceClusterWeight(\lambda,K)\)foreach\(z_{i}\in K\)do
5\(P_{Z_{i}}=prob(z_{i},d_{t})\) using Eq. (3)
6 end foreach
7\(i=\arg\max\limits_{i}(P_{Z_{i}})\)
8\(P_{Z_{n}}=\) calculate the probability of new cluster using Eq. (6)
9if\(P_{Z_{i}}<P_{Z_{n}}\)then
10\(m_{z_{n}}=1\)
11\(n_{z_{n}}^{w}=N_{d_{t}}^{w}\)
12\(cw_{z_{n}}=cw_{d_{t}}\)
13\(len_{z_{n}}=len_{d_{t}}\)
14\(l_{z_{n}}=1,u_{z_{n}}=t\)
15\(K=K\cup z_{n}\)
16else
17\(m_{z_{i}}=m_{z_{i}}+1\)
18\(n_{z_{i}}^{w}=n_{z_{i}}^{w}+N_{dt}^{w}\)
19\(cw_{z_{i}}=cw_{z_{i}}\cup cw_{d_{t}}\)
20\(len_{z_{i}}=len_{z_{i}}+len_{d_{t}}\)
21\(l_{z_{i}}=1\), \(u_{z_{i}}=t\)
22
23 end while
24
25 end for
```

**Algorithm 1**OSDM

The desirable addition property of cluster feature allows updating each micro-cluster in an online way.

**Definition 1:** A document \(d\) can be added to a cluster \(z\) by using the _addition property_.

\begin{tabular}{r l} \(m_{z}=m_{z}+1\) & \\  & \(n_{z}^{w}=n_{z}^{w}+N_{d}^{w}\) & \(\forall w\in d\) \\  & \(cw_{z}=cw_{z}\cup cw_{d}\) & \\ \end{tabular}

\begin{tabular}{r l} \multicolumn{2}{l}{\(len_{z}=len_{z}+len_{d}\)} \\ \multicolumn{2}{l}{Here, \(cw_{d}\) is word to word co-occurrence of the document, and \(len_{d}\) represents the number of total words in the document. The complexity of updating a cluster by adding a document is \(O(\mathcal{L})\), where \(\mathcal{L}\) is the average length of the document. This property is useful to update evolving micro-clusters in the text stream clustering procedure.

### OSDM Algorithm

We propose a semantic-enhanced non-parametric dirichlet model to cluster the short text streams in an online way, called OSDM. The proposed algorithm allows processing each instance incrementally and updates the model accordingly.

The procedure of OSDM is given in Algorithm 1. Initially, it creates a new cluster for the first doc 

ument and the document is assigned to the newly created CF set. Afterward, each arriving document in the stream either choose an existing cluster or generate a new cluster. The corresponding probability for choosing either of an existing cluster or a new cluster is computed using Equation (6) and (3), respectively. The CF vector with the highest probability is updated using the addition property.

To deal with the cluster evolution (i.e., evolving topics) in text streams, many existing approaches often delete the old clusters by using some of the forgetting mechanisms (e.g., decay rate) [20; 19]. Instead of deleting old clusters, MSTreamF [75] deletes old batches. In this study, we investigate the importance of each micro-cluster to handle the cluster evolution problem. Specifically, the importance of each micro-cluster is decreased over time if it is not updated. \(l_{z}\) in CF stores weight of each cluster. If the weight is approximately equals to zero, then the cluster is removed from the model, i.e., it cannot capture recent topics in the text stream. For this purpose, we applied the exponential decay function, \(l_{z}=l_{z}\times 2^{-\lambda\times(\triangle t)}\). Here, \(\triangle t\) is the elapsed time from the last update, and \(\lambda\) is the decay rate. The decay rate must be adjusted depending upon the applications at hand. The initial value of \(l_{z}\) (See Line 16 of Algorithm 1) is set to 1. Afterward, the importance of micro-cluster is exponentially decreases over time. We can also store the deleted clusters in a permanent disk for offline analysis.

**Complexity Analysis.** The OSDM algorithm always maintains the average \(\bar{K}\) number of current topics (CF sets). Every CF set store average \(\bar{V}\) number of words in \(n_{z}^{w}\) and at most \(|\bar{V}_{z}|\times|\bar{V}_{z}|\) in \(cw_{z}\). Thus the space complexity of OSDM is \(O(\bar{K}(\bar{V}+\bar{V}^{2})+VD)\), where \(V\) is the size of active vocabulary and \(D\) is the number of active documents. On other side, OSDM calculates the probability of arriving document with each cluster (see Line 6 of Algorithm 1). Therefore, the time complexity of OSDM is \(O(\bar{K}(\mathcal{L}\bar{V}))\), where \(\mathcal{L}\) is the average size of arriving document.

## 5 Experimental Study

### Datasets and evaluation metrics

To evaluate the performance of the proposed algorithm, we conduct experiments on three real and two synthetic datasets. These datasets were also used in [74, 8, 19, 18, 17, 16] to evaluate short text clustering models. In the preprocessing step, we removed stop words, converted all text into lowercase, and stemming. The description of the datasets is as follows.

* **News (Ns):** This dataset is collected by [74], which contains 11,109 news title belong to 152 topics.
* **Reuters (Rs):** Similar to [74] we skip the documents with more than one class and obtained the dataset consists of 9,447 documents from 66 topics.
* **Tweets (Ts):** This dataset contain 30,322 tweets which are relevant to 269 topics in the TREC 3 microblog. Footnote 3: [http://trec.nist.gov/data/microblog.html](http://trec.nist.gov/data/microblog.html)
* **News-T (Ns-T) and Reuters-T (Rs-T):** Naturally, we may find a situation where topics in social media appear only for a certain time period and then disappear. However, the documents of each topic in original dataset is observed for long period of time. Therefore, to construct synthetic dataset we sorted documents datasets by topic in two datasets including _Reuters_ and _News_. After sorting, we then divide each dataset into sixteen equal chunks and shuffled them.

We adopted five different evaluation metrics for deep analysis of all algorithms, which include Normalized Mutual Information (NMI), Homogeneity (Ho.), V-Measure (VM), Accuracy (Acc.) and cluster Purity (Pur.). We utilized sklearn4 API to implement these metrics. We compute the measures on overall clustering results [74]. Homogeneity measures that each cluster should have only members of a single class. Whereas, V-measure calculates how successfully the criteria of completeness and homogeneity are satisfied. Cluster purity measures the true positive instances in each cluster. The typical NMI measure calculates the overall clustering quality.

Footnote 4: [http://scikit-learn.org](http://scikit-learn.org)

### Baselines

We have selected four state-of-the-art representative algorithms for stream text clustering to com 

pare OSDM (Os). A brief description of these algorithms are given as follows.

1. **DTM**[20] is an extension of Latent Dirichlet Allocation which traces the evolution of hidden topics from corpus over time. It was designed to deal with the sequential documents.
2. **Sumblr (Sb)**[34] is an online stream clustering algorithm for tweets. With only one pass, it enables the model to cluster the tweets efficiently while maintaining cluster statistics.
3. **DMM**[26] is a Dirichlet multinomial mixture model for short text clustering, which does not consider temporal dependency of instances.
4. **MStreamF**[27] is the latest model to deal with infinite number of latent topics in short text while processing one batch at a time. Two models of MStreamF were proposed, one with one-pass clustering process, and another with gibbs sampling. We refer to the former algorithm as MStreamF-O (MF-O) and the latter as MStreamF-G (MF-G).

We try to find the optimal parameter values of all baseline algorithms with grid search. Finally, we set \(\alpha=0.01\) for DTM, \(\beta=0.02\) for Sumblr. For MStreamF-O and MStreamF-G, we set \(\alpha=0.03\) and \(\beta=0.03\). As defined in Yin et al. (2018), we set the number of iterations to 10 and \(saved{batches}=2\) for MStreamF-G. We set \(\alpha=0.3\) and \(\beta=0.3\) for DMM. The DTM, DMM and Sumblr needs fixed number of cluster as input therefore we set \(K=300,K=170\) and \(K=80\) for Tweets, News and Reuters datasets, respectively. We set \(\alpha=2e^{-3}\), \(\beta=4e^{-5}\) and \(\lambda=6e^{-6}\) for OSDM. The source code of OSDM is publicly available at: [https://github.com/JayKumarrfOSDM](https://github.com/JayKumarrfOSDM).

### Comparison with state-of-the-art methods

In this section, we provide a detailed comparative analysis of OSDM with state-of-the-art algorithms. The overall results are summarized in Table 1. We report NMI, Homogeneity, v-measure, purity and accuracy of each algorithm. Additionally, we also evaluate the performance of each algorithm over different time-stamps of the stream (see Figure 2). Further, we studied the parameter sensitivity and runtime of OSDM, respectively.

From Table 1, we can see that OSDM outperformed all baseline algorithms on almost every dataset in terms of all measures. Here, MStreamF-G yielded much better results on the Ns-T data in terms of NMI measure. The reason behind might be the multiple iterations of each batch in the stream. However, MStreamF-G requires more execution time to process the data. In contrast, our proposed algorithm OSDM processes the data only once. And we can also observe that OSDM achieves the highest NMI in other data sets. In addition, the crucial part of evaluating the cluster similarity is measured by the homogeneity measure. We can see that OSDM outperformed all previous algorithms. It also shows the same statistics except for v-measure of DTM. Likewise, our model generates more pure clusters. Furthermore, to investigate the performance over time, we plot the performance of

Figure 2: The performance of different text steam clustering algorithm over time (in thousand points) in terms of NMI measure.

 

all algorithms over time in Figure 2.

### Sensitivity Analysis

We perform sensitivity analysis for OSDM with respects to three input parameters: concentration parameter \(\alpha\), \(\beta\), and decay function parameter \(\lambda\) on the _Tweets_ dataset. From Figure 3a, we can observe the effect of \(\alpha\), which ranges from \(9e^{-3}\) to \(9e^{-1}\). The performance in terms of all evaluation measures is stable over the different values of parameters. The \(\alpha\) parameter is responsible for finer clustering, that is why we can observe a little fluctuation in initial values. Figure 3b shows the performance on different values of \(\beta\), which ranges from \(1e^{-4}\) to \(1e^{-2}\). As we already defined that we modified homogeneity part of the clustering model (see Equation (3)), and \(\beta\) is the related hyper-parameter. We can observe that after a certain range, the values of all the evaluation measure become stable. The crucial point to be observed is the stability of homogeneity on different values of \(\beta\). Figure 3c shows effect of \(\lambda\) ranges from \(9e^{-4}\) to \(9e^{-6}\). Our model follows the forgetting mechanism on decay factor \(\lambda\) and the clusters are deleted from model when the value is approximately equals to zero. We can observe the performance of OSDM

Figure 4: The runtime of different text stream clustering algorithms.

\begin{table}
\begin{tabular}{l r r r r r r} \hline  & \multicolumn{5}{c}{**Datasets**} \\ \cline{2-7}
**Alg.** & **Eva.** & Ns & Ts & Rs & Nes-T & Rs-T \\ \hline OSDM & **0.815** & **0.836** & **0.552** & 0.858 & **0.554** \\ MF-O & 0.685 & 0.746 & 0.361 & 0.803 & 0.381 \\ MF-G & NMI & 0.780 & 0.795 & 0.364 & **0.888** & 0.405 \\ Sb & 0.575 & 0.698 & 0.464 & 0.723 & 0.494 \\ DTM & 0.808 & 0.800 & 0.537 & 0.810 & 0.537 \\ DMM & 0.586 & 0.636 & 0.448 & 0.582 & 0.476 \\ \hline OSDM & **0.951** & **0.936** & **0.954** & **0.900** & **0.964** \\ MF-O & 0.654 & 0.695 & 0.374 & 0.778 & 0.385 \\ MF-G & Ho. & 0.751 & 0.738 & 0.319 & 0.900 & 0.343 \\ Sb & 0.547 & 0.758 & 0.402 & 0.747 & 0.574 \\ DTM & 0.833 & 0.822 & 0.659 & 0.837 & 0.657 \\ DMM & 0.588 & 0.622 & 0.466 & 0.565 & 0.497 \\ \hline OSDM & 0.805 & **0.831** & **0.479** & 0.857 & 0.478 \\ MF-O & 0.684 & 0.744 & 0.361 & 0.803 & 0.380 \\ MF-G & VM & 0.779 & 0.793 & 0.361 & **0.888** & 0.400 \\ Sb & 0.575 & 0.696 & 0.458 & 0.723 & 0.436 \\ DTM & **0.808** & 0.800 & 0.526 & 0.810 & **0.527** \\ DMM & 0.586 & 0.636 & 0.448 & 0.582 & 0.476 \\ \hline OSDM & **0.907** & **0.890** & **0.962** & **0.851** & **0.972** \\ MF-O & 0.552 & 0.529 & 0.602 & 0.636 & 0.608 \\ MF-G & Pur. & 0.653 & 0.801 & 0.530 & 0.835 & 0.606 \\ Sb & 0.414 & 0.609 & 0.609 & 0.580 & 0.770 \\ DTM & 0.767 & 0.749 & 0.793 & 0.765 & 0.795 \\ DMM & 0.456 & 0.473 & 0.673 & 0.398 & 0.694 \\ \hline OSDM & **0.880** & 0.665 & **0.927** & **0.769** & **0.952** \\ MF-O & 0.420 & 0.246 & 0.577 & 0.584 & 0.447 \\ MF-G & Acc. & 0.517 & **0.707** & 0.452 & 0.606 & 0.461 \\ Sb & 0.606 & 0.539 & 0.652 & 0.653 & 0.620 \\ DTM & 0.647 & 0.246 & 0.669 & 0.294 & 0.644 \\ DMM & 0.334 & 0.150 & 0.649 & 0.073 & 0.500 \\ \hline \end{tabular}
\end{table}
Table 1: The performance of different algorithms on five data sets in terms of different measures including Mutual Information (NMI), Homogeneity (Ho.), V-Measure (VM), Accuracy (Acc.) and cluster Purity (Pur.).

Figure 3: The sensitivity analysis with different parameters, including \(\alpha,\beta\) and \(\lambda\).

 

on different decay factors. It can be observed that the behavior of a given evaluation measure is stable over time.

### Runtime

To compare the runtime of different algorithms, we performed all experiments on a PC with _core i5-3470_ and 8GB memory. Figure 4 shows the runtime of all algorithms on the tweets dataset. We can observe that Sumblr required the highest execution time to cluster the instances. Whereas, the runtime of other algorithms are comparable. Due to simple execution process of each instance MSTreamF-O took least time because it does not need to maintain semantic similarity. Comparatively, MSTreamF-G required much higher time than OSDM. The reason is that it needs to execute each batch data multiple times. Due to online nature, the overall speed of OSDM is more efficient than most existing algorithms, and the benefit is strengthened with more and more arriving instances.

## 6 Conclusion

In this paper, we propose a new online semantic-enhanced dirichlet model for short text stream clustering. In contrast to existing approaches, OSDM does not require to specify the batch size and the dynamic number evolving clusters. It dynamically assigns each arriving document into an existing cluster or generating a new cluster based on the poly un scheme. More importantly, OSDM tried to incorporate semantic information in the proposed graphical representation model to remove the term ambiguity problem in short-text clustering. Building upon the semantic embedding and online learning, our method allows finding high-quality evolving clusters. Extensive results further demonstrate that OSDM has better performance compared to many state-of-the-art algorithms.

## Acknowledgments

This work is supported by the National Natural Science Foundation of China (61976044), Fundamental Research Funds for the Central Universities (ZYGK2019Z014), Fok Ying-Tong Education Foundation for Young Teachers in the Higher Education Institutions of China (161062), National key research and development program (2016YFB0502300).

## References

* C. Aggarwal (2018)A survey of stream clustering algorithms. In Data Clustering, pp. 231-258. Cited by: SS1.
* C. Aygarwal et al. (2003)A framework for clustering evolving data streams. In International conference on Very large data bases, pp. 81-92. Cited by: SS1.
* D. C. Aggarwal and P. S. Yu (2010)On clustering massive text and categorical data streams. Knowledge and Information Systems24 (2), pp. 171-196. Cited by: SS1.
* A. Ahmed and E. P. Xing (2008)Dynamic non-parametric mixture models and the recurrent chinese restaurant process: with applications to evolutionary clustering. In Proceedings of SIAM International Conference on Data Mining, pp. 219-230. Cited by: SS1.
* H. Amoualian, M. Clausel, E. Gaussier, M. Amini, G. Clausel, M. Amini, E. Gaussier, and M. Amini (2016)Streaming-LDA: a copula-based approach to modeling topic dependencies in document streams. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 695-704. Cited by: SS1.
* D. Blackwell, J. B. MacQueen, Others, and D. R. Brillinger (1973)Ferguson distributions via folya um schemes. The annals of statistics1 (2), pp. 353-355. Cited by: SS1.
* D. M. Blei and J. D. Lafferty (2006)Dynamic topic models. ACM International Conference Proceeding Series148, pp. 113-120. Cited by: SS1.
* D. M. Blei, A. Y. Ng, and M. I. Jordan (2003)Latent dirichlet allocation. Journal of machine Learning research3, pp. 993-1022. Cited by: SS1.
* J. Chen, Z. Gong, and W. Liu (2019)A nonparametric model for online topic discovery with word embeddings. Information Sciences504, pp. 32-47. Cited by: SS1.
* S. Ud Din and J. Shao (2020)Exploiting evolving micro-clusters for data stream classification with emerging class detection. Information Sciences507, pp. 404-420. Cited by: SS1.
* T. S. Ferguson and T. S. Ferguson (1973)A bayesian analysis of some nonparametric problems. The annals of statistics1 (2), pp. 209-230.

 

Hongyu Gong, Tarek Sakakini, Suma Bhat, and Jinjun Xiong. 2018. Document similarity for texts of varying lengths via hidden topics. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_, volume 1, pages 2341-2351.
* Hadifar et al. (2019) Amir Hadifar, Lucas Sterckx, Thomas Demeester, and Chris Develder. 2019. A Self-Training Approach for Short Text Clustering. In _Proceedings of the 4th Workshop on Representation Learning for NLP (ACL)_, pages 194-199.
* Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. _Association for Computational Linguistics_, 1:328-339.
* Huang et al. (2013) Ruizhang Rui-zhang Huang, Guan Yu, Zhaojun Wang, Jun Zhang, and Liangxing Shi. 2013. Dirichlet process mixture model for document clustering with feature partition. _IEEE Transactions on Knowledge and Data Engineering_, 25(8):1748-1759.
* Islam et al. (2019) Md. Kamrul Islam, Md. Manjur Ahmed, and Kamal Z. Zamli. 2019. A buffer-based online clustering for evolving data stream. _Information Sciences_, 489:113-135.
* Jia et al. (2018) Caiyan Jia, Matthew B. Carson, Xiaoyang Wang, and Jian Yu. 2018. Concept decompositions for short text clustering by identifying word communities. _Pattern Recognition_, 76:691-703.
* Liang et al. (2016) Shangsong Liang, Emine Yilmaz, and Evangelos Kanoulas. 2016. Dynamic Clustering of Streaming Short Documents. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, pages 995-1004. ACM.
* Rahdisahiaji (2009) Alireza Rezaei Mahdiraji. 2009. Clustering data stream: A survey of algorithms. _International Journal of Knowledge-Based and Intelligent Engineering Systems_, 13(2):39-44.
* Neal (2000) Radford M Neal. 2000. Markov chain sampling methods for Dirichlet process mixture models. _Journal of computational and graphical statistics_, 9(2):249-265.
* Nguyen et al. (2015) Hai Long Nguyen, Yew Kwong Woon, and Wee Keong Ng. 2015. A survey on data stream clustering and classification. _Knowledge and Information Systems_, 45(3):535-569.
* Qiang et al. (2018) Jipeng Qiang, Yun Li, Yunhao Yuan, and Xindong Wu. 2018. Short text clustering based on Pitman-Yor process mixture model. _Applied Intelligence_, 48(7):1802-1812.
* Shao et al. (2019) Junming Shao, Yue Tan, Lianli Gao, Qinli Yang, Claudia Plant, and Ira Assent. 2019. Synchronization-based clustering on evolving data stream. _Inf. Sci._, 501:573-587.
* Zhou et al. (2013) Lidan Zhou, Zhenhua Wang, Ke Chen, and Gang Chen. 2013. Sumblr Continuous Summarization of Evolving Tweet Streams. In _Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval_, pages 533-542. ACM.
* Silva et al. (2013) Jonathan A. Silva, Elaine R. Faria, Rodrigo C. Barros, Eduardo R. Hruschka, Andre C. P. L. F. de Carvalho, and Joao Gama. 2013. Data stream clustering: A Survey. _ACM Computing Surveys_, 46(1):1-31.
* Teh et al. (2006) Yee Whye Teh, Michael I. Jordan, Matthew J. Beal, and David M. Blei. 2006. Hierarchical Drichlet processes. _Journal of the American Statistical Association_, 101(476):1566-1581.
* Wang et al. (2012) Yu Wang, Eugene Agichtein, and Michele Benzi. 2012. TM-LDA: Efficient Online Modeling of Latent Topic Transitions in Social Media. In _International conference on Knowledge discovery and data mining_, pages 123-131. ACM.
* Wei et al. (2007) Xing Wei, Jimeng Sun, and Xuerui Wang. 2007. Dynamic mixture models for multiple time series. In _International Joint Conference on Artificial Intelligence_, pages 2909-2914.
* Yin et al. (2018) Jianhua Yin, Daren Chao, Zhongkun Liu, Wei Zhang, Xiaohui Yu, and Jianyong Wang. 2018. Model-based Clustering of Short Text Streams. In _ACM International Conference on Knowledge Discovery and Data Mining_, pages 2634-2642.
* Yin and Wang (2014) Jianhua Yin and Jianyong Wang. 2014. A dirichlet multinomial mixture model-based approach for short text clustering. In _Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, pages 233-242. ACM.
* Yin and Wang (2016a) Jianhua Yin and Jianyong Wang. 2016a. A model-based approach for text clustering with outlier detection. In _IEEE International Conference on Data Engineering_, pages 625-636.
* Yin and Wang (2016b) Jianhua Yin and Jianyong Wang. 2016b. A Text Clustering Algorithm Using an Online Clustering Scheme for Initialization. In _ACM International Conference on Knowledge Discovery and Data Mining_, pages 1995-2004.
* 24th International Conference, DASFAA 2019, Chiang Mai, Thailand, April 22-25, 2019, Proceedings, Part I_, volume 11446 of _Lecture Notes in Computer Science_, pages 276-291. Springer.
* Zhao et al. (2016) Yukun Zhao, Shangsong Liang, Zhaochun Ren, Jun Ma, Emine Yilmaz, and Maarten de Rijke. 2016. Explainable User Clustering in Short Text Streams. In _International ACM conference on Research and Development in Information Retrieval_, pages 155-164.

 

Shi Zhong. 2005. Efficient streaming text clustering.
* [18]