SlowGuess/ACL_2020_OCR huggingface dataset:
- News (Ns): This dataset is collected by (Yin and Wang, 2014), which contains 11,109 news title belong to 152 topics.
- Reuters (Rs): Similar to (Yin and Wang, 2016b) we skip the documents with more than one class and obtained the dataset consists of 9,447 documents from 66 topics.
- Tweets (Ts): This dataset contains 30,322 tweets which are relevant to 269 topics in the TREC microblog.
- News-T (Ns-T) and Reuters-T (Rs-T): Naturally, we may find a situation where topics in social media appear only for a certain time period and then disappear. However, the documents of each topic in original dataset is observed for long period of time. Therefore, to construct synthetic dataset we sorted documents datasets by topic in two datasets including Reuters and News. After sorting, we then divide each dataset into sixteen equal chunks and shuffled them.
We adopted five different evaluation metrics for deep analysis of all algorithms, which include Normalized Mutual Information (NMI), Homogeneity (Ho.), V-Measure (VM), Accuracy (Acc.) and cluster Purity (Pur.). We utilized sklearn<sup>4</sup> API to implement these metrics. We compute the measures on overall clustering results (Yin and Wang, 2014). Homogeneity measures that each cluster should have only members of a single class. Whereas, V-measure calculates how successfully the criteria of completeness and homogeneity are satisfied. Cluster purity measures the true positive instances in each cluster. The typical NMI measure calculates the overall clustering quality.
We have selected four state-of-the-art representative algorithms for stream text clustering to com
3http://trec.nist.gov/data/microblog.html
<sup>4</sup>http://scikit-learn.org
pare OSDM (Os). A brief description of these algorithms are given as follows.
(1) DTM (Blei and Lafferty, 2006) is an extension of Latent Dirichlet Allocation which traces the evolution of hidden topics from corpus over time. It was designed to deal with the sequential documents.
(2) Sumblr (Sb) (Shou et al., 2013) is an online stream clustering algorithm for tweets. With only one pass, it enables the model to cluster the tweets efficiently while maintaining cluster statistics.
(3) DMM (Yin and Wang, 2014) is a Dirichlet multinomial mixture model for short text clustering, which does not consider temporal dependency of instances.
(4) MStreamF (Yin et al., 2018) is the latest model to deal with infinite number of latent topics in short text while processing one batch at a time. Two models of MStreamF were proposed, one with one-pass clustering process, and another with gibbs sampling. We refer to the former algorithm as MStreamF-O (MF-O) and the latter as MStreamF-G (MF-G).
We try to find the optimal parameter values of all baseline algorithms with grid search. Finally, we set  $\alpha = 0.01$  for DTM,  $\beta = 0.02$  for Sumblr. For MStreamF-O and MStreamF-G, we set  $\alpha = 0.03$  and  $\beta = 0.03$ . As defined in (Yin et al., 2018), we set the number of iterations to 10 and saved-batches  $= 2$  for MStreamF-G. We set  $\alpha = 0.3$  and  $\beta = 0.3$  for DMM. The DTM, DMM and Sumblr needs fixed number of cluster as input therefore we set  $K = 300$ ,  $K = 170$  and  $K = 80$  for Tweets, News and Reuters datasets, respectively. We set  $\alpha = 2e^{-3}$ ,  $\beta = 4e^{-5}$  and  $\lambda = 6e^{-6}$  for OSDM. The source code of OSDM is publicly available at: https://github.com/JayKumarr/OSDM.
5.3 Comparison with state-of-the-art methods
In this section, we provide a detailed comparative analysis of OSDM with state-of-the-art algorithms. The overall results are summarized in Table 1. We report NMI, Homogeneity, v-measure, purity and accuracy of each algorithm. Additionally, we also evaluate the performance of each algorithm over different time-stamps of the stream (see Figure 2).
(a) News
(b) News-T
(c)Reuters
(d)Reuters-T
(e) Tweets
Figure 2: The performance of different text steam clustering algorithm over time (in thousand points) in terms of NMI measure.
Further, we studied the parameter sensitivity and runtime of OSDM, respectively.
From Table 1, we can see that OSDM outperformed all baseline algorithms on almost every dataset in terms of all measures. Here, MStreamFG yielded much better results on the Ns-T data in terms of NMI measure. The reason behind might be the multiple iterations of each batch in the stream. However, MStreamFG requires more execution time to process the data. In contrast, our proposed algorithm OSDM processes the data only once. And we can also observe that OSDM achieves the highest NMI in other data sets. In addition, the crucial part of evaluating the cluster similarity is measured by the homogeneity measure. We can see that OSDM outperformed all previous algorithms. It also shows the same statistics except for v-measure of DTM. Likewise, our model generates more pure clusters. Furthermore, to investigate the performance over time, we plot the performance of
(b)  $\beta$
(c)  $\lambda$
Figure 3: The sensitivity analysis with different parameters, including  $\alpha$ ,  $\beta$  and  $\lambda$ .
Alg.	Eva.	Datasets
Ns	Ts	Rs	Ns-T	Rs-T
OSDM	0.815	0.836	0.552	0.858	0.554
MF-O	0.685	0.746	0.361	0.803	0.381
MF-G	NMI	0.780	0.795	0.364	0.888	0.405
Sb	0.575	0.698	0.464	0.723	0.494
DTM	0.808	0.800	0.537	0.810	0.537
DMM	0.586	0.636	0.448	0.582	0.476
OSDM	0.951	0.936	0.954	0.900	0.964
MF-O	0.654	0.695	0.374	0.778	0.385
MF-G	Ho.	0.751	0.738	0.319	0.900	0.343
Sb	0.547	0.758	0.402	0.747	0.574
DTM	0.833	0.822	0.659	0.837	0.657
DMM	0.588	0.622	0.466	0.565	0.497
OSDM	0.805	0.831	0.479	0.857	0.478
MF-O	0.684	0.744	0.361	0.803	0.380
MF-G	VM	0.779	0.793	0.361	0.888	0.400
Sb	0.575	0.696	0.458	0.723	0.436
DTM	0.808	0.800	0.526	0.810	0.527
DMM	0.586	0.636	0.448	0.582	0.476
OSDM	0.907	0.890	0.962	0.851	0.972
MF-O	0.552	0.529	0.602	0.636	0.608
MF-G	Pur.	0.653	0.801	0.530	0.835	0.606
Sb	0.414	0.609	0.609	0.580	0.770
DTM	0.767	0.749	0.793	0.765	0.795
DMM	0.456	0.473	0.673	0.398	0.694
OSDM	0.880	0.665	0.927	0.769	0.952
MF-O	0.420	0.246	0.577	0.584	0.447
MF-G	Acc.	0.517	0.707	0.452	0.606	0.461
Sb	0.606	0.539	0.652	0.653	0.620
DTM	0.647	0.246	0.669	0.294	0.644
DMM	0.334	0.150	0.649	0.073	0.500
Table 1: The performance of different algorithms on five data sets in terms of different measures including Mutual Information (NMI), Homogeneity (Ho.), V-Measure (VM), Accuracy (Acc.) and cluster Purity (Pur.).
all algorithms over time in Figure 2.
We perform sensitivity analysis for OSDM with respects to three input parameters: concentration parameter  $\alpha$ ,  $\beta$ , and decay function parameter  $\lambda$  on the Tweets dataset. From Figure 3a, we can observe the effect of  $\alpha$ , which ranges from  $9e^{-3}$  to  $9e^{-1}$ . The performance in terms of all evaluation measures is stable over the different values of parameters. The  $\alpha$  parameter is responsible for finer clustering, that is why we can observe a little fluctuation in initial values. Figure 3b shows the performance on different values of  $\beta$ , which ranges from  $1e^{-4}$  to  $1e^{-2}$ . As we already defined that we modified homogeneity part of the clustering model (see Equation (3)), and  $\beta$  is the related hyper-parameter. We can observe that after a certain range, the values of all the evaluation measure become stable. The crucial point to be observed is the stability of homogeneity on different values of  $\beta$ . Figure 3c shows effect of  $\lambda$  ranges from  $9e^{-4}$  to  $9e^{-6}$ . Our model follows the forgetting mechanism on decay factor  $\lambda$  and the clusters are deleted from model when the value is approximately equals to zero. We can observe the performance of OSDM
Figure 4: The runtime of different text stream clustering algorithms.
on different decay factors. It can be observed that the behavior of a given evaluation measure is stable over time.
5.5 Runtime
To compare the runtime of different algorithms, we performed all experiments on a PC with core i5-3470 and 8GB memory. Figure 4 shows the runtime of all algorithms on the tweets dataset. We can observe that Sumblr required the highest execution time to cluster the instances. Whereas, the runtime of other algorithms are comparable. Due to simple execution process of each instance MStreamF-O took least time because it does not need to maintain semantic similarity. Comparatively, MStreamF-G required much higher time than OSDM. The reason is that it needs to execute each batch data multiple times. Due to online nature, the overall speed of OSDM is more efficient than most existing algorithms, and the benefit is strengthened with more and more arriving instances.
In this paper, we propose a new online semantic-enhanced dirichlet model for short text stream clustering. In contrast to existing approaches, OSDM does not require to specify the batch size and the dynamic number evolving clusters. It dynamically assigns each arriving document into an existing cluster or generating a new cluster based on the poly urn scheme. More importantly, OSDM tried to incorporate semantic information in the proposed graphical representation model to remove the term ambiguity problem in short-text clustering. Building upon the semantic embedding and online learning, our method allows finding high-quality evolving clusters. Extensive results further demonstrate that OSDM has better performance compared to many state-of-the-art algorithms.
This work is supported by the National Natural Science Foundation of China (61976044), Fundamental Research Funds for the Central Universities (ZYGX2019Z014), Fok Ying-Tong Education Foundation for Young Teachers in the Higher Education Institutions of China (161062), National key research and development program (2016YFB0502300).
Charu C. Aggarwal. 2018. A survey of stream clustering algorithms. In Data Clustering, pages 231-258.
Charu C Aggarwal, Jiawei Han, Jianyong Wang, Philip S. Yu, Jianyong Wang Jiawei Han, Philip S. Yu, Jiawei Han, Jianyong Wang, and Philip S. Yu. 2003. A Framework for Clustering Evolving Data Streams. In International conference on Very large data bases, pages 81-92.
Charu C Aggarwal and Philip S. Yu. 2010. On Clustering Massive Text and Categorical Data Streams. Knowledge and Information Systems, 24(2):171-196.
Amr Ahmed and Eric P Xing. 2008. Dynamic Non-Parametric Mixture Models and The Recurrent Chinese Restaurant Process: with Applications to Evolutionary Clustering. In Proceedings of SIAM International Conference on Data Mining, pages 219-230.
Hesam Amoualian, Marianne Clausel, Eric Gaussier, Massih-Reza Amini, Marianne Clausel, Massih-Reza Amini, Éric Gaussier, and Massih-Reza Amini. 2016. Streaming-LDA: A Copula-based Approach to Modeling Topic Dependencies in Document Streams. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 695-704. ACM.
David Blackwell, James B MacQueen, Others, and David R. Brillinger. 1973. Ferguson distributions via Pólya urn schemes. The annals of statistics, 1(2):353-355.
David M. Blei and John D. Lafferty. 2006. Dynamic topic models. ACM International Conference Proceeding Series, 148:113-120.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3:993-1022.
Junyang Chen, Zhiguo Gong, and Weiwen Liu. 2019. A nonparametric model for online topic discovery with word embeddings. Information Sciences, 504:32-47.
Salah Ud Din and Junming Shao. 2020. Exploiting evolving micro-clusters for data stream classification with emerging class detection. Information Sciences, 507:404-420.
Salah Ud Din, Junming Shao, Jay Kumar, Waqar Ali, Ji-aming Liu, and Yu Ye. 2020. Online Reliable Semi-supervised Learning on Evolving Data Streams. Information Sciences, 507.
Thomas S Ferguson and Thomas S Ferguson. 1973. A Bayesian analysis of some nonparametric problems. The annals of statistics, 1(2):209-230.
Hongyu Gong, Tarek Sakakini, Suma Bhat, and Jinjun Xiong. 2018. Document similarity for texts of varying lengths via hidden topics. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, volume 1, pages 2341-2351.
Amir Hadifar, Lucas Sterckx, Thomas Demeester, and Chris Develder. 2019. A Self-Training Approach for Short Text Clustering. In Proceedings of the 4th Workshop on Representation Learning for NLP (ACL), pages 194-199.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. Association for Computational Linguistics, 1:328-339.
Ruizhang Rui-zhang Huang, Guan Yu, Zhaojun Wang, Jun Zhang, and Liangxing Shi. 2013. Dirichlet process mixture model for document clustering with feature partition. IEEE Transactions on Knowledge and Data Engineering, 25(8):1748-1759.
Md. Kamrul Islam, Md. Manjur Ahmed, and Kamal Z. Zamli. 2019. A buffer-based online clustering for evolving data stream. Information Sciences, 489:113-135.
Caiyan Jia, Matthew B. Carson, Xiaoyang Wang, and Jian Yu. 2018. Concept decompositions for short text clustering by identifying word communities. Pattern Recognition, 76:691-703.
Shangsong Liang, Emine Yilmaz, and Evangelos Kanoulas. 2016. Dynamic Clustering of Streaming Short Documents. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 995-1004. ACM.
Alireza Rezaei Mahdiraji. 2009. Clustering data stream: A survey of algorithms. International Journal of Knowledge-Based and Intelligent Engineering Systems, 13(2):39-44.
Radford M Neal. 2000. Markov chain sampling methods for Dirichlet process mixture models. Journal of computational and graphical statistics, 9(2):249-265.
Hai Long Nguyen, Yew Kwong Woon, and Wee Keong Ng. 2015. A survey on data stream clustering and classification. Knowledge and Information Systems, 45(3):535-569.
Jipeng Qiang, Yun Li, Yunhao Yuan, and Xindong Wu. 2018. Short text clustering based on Pitman-Yor process mixture model. Applied Intelligence, 48(7):1802-1812.
Junming Shao, Yue Tan, Lianli Gao, Qinli Yang, Claudia Plant, and Ira Assent. 2019. Synchronization-based clustering on evolving data stream. Inf. Sci., 501:573-587.
Lidan Shou, Zhenhua Wang, Ke Chen, and Gang Chen. 2013. Sumblr Continuous Summarization of Evolving Tweet Streams. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, pages 533-542. ACM.
Jonathan A. Silva, Elaine R. Faria, Rodrigo C. Barros, Eduardo R. Hruschka, André C. P. L. F. de Carvalho, and João Gama. 2013. Data stream clustering: A Survey. ACM Computing Surveys, 46(1):1-31.
Yee Whye Teh, Michael I. Jordan, Matthew J. Beal, and David M. Blei. 2006. Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101(476):1566-1581.
Yu Wang, Eugene Agichtein, and Michele Benzi. 2012. TM-LDA: Efficient Online Modeling of Latent Topic Transitions in Social Media. In International conference on Knowledge discovery and data mining, pages 123-131. ACM.
Xing Wei, Jimeng Sun, and Xuerui Wang. 2007. Dynamic mixture models for multiple time series. In International Joint Conference on Artificial Intelligence, pages 2909-2914.
Jianhua Yin, Daren Chao, Zhongkun Liu, Wei Zhang, Xiaohui Yu, and Jianyong Wang. 2018. Model-based Clustering of Short Text Streams. In ACM International Conference on Knowledge Discovery and Data Mining, pages 2634-2642.
Jianhua Yin and Jianyong Wang. 2014. A dirichlet multinomial mixture model-based approach for short text clustering. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 233-242. ACM.
Jianhua Yin and Jianyong Wang. 2016a. A model-based approach for text clustering with outlier detection. In IEEE International Conference on Data Engineering, pages 625-636.
Jianhua Yin and Jianyong Wang. 2016b. A Text Clustering Algorithm Using an Online Clustering Scheme for Initialization. In ACM International Conference on Knowledge Discovery and Data Mining, pages 1995-2004.
Zhong Zhang, Chongming Gao, Chongzhi Liu, Qinli Yang, and Junming Shao. 2019. Towards robust arbitrarily oriented subspace clustering. In Database Systems for Advanced Applications - 24th International Conference, DASFAA 2019, Chiang Mai, Thailand, April 22-25, 2019, Proceedings, Part I, volume 11446 of Lecture Notes in Computer Science, pages 276-291. Springer.
Yukun Zhao, Shangsong Liang, Zhaochun Ren, Jun Ma, Emine Yilmaz, and Maarten de Rijke. 2016. Explainable User Clustering in Short Text Streams. In International ACM conference on Research and Development in Information Retrieval, pages 155-164.
Shi Zhong. 2005. Efficient streaming text clustering. Neural Networks, 18(5-6):790-798.