SlowGuess/ACL_2020_OCR huggingface dataset:
Jay Kumar and Junming Shao* and Salah ud din
Data Mining Lab, School of Computer Science and Engineering,
University of Electronic Science and Technology of China, Chengdu, 611731, China
jay_tharwani1992@yahoo.com, junmshao@uestc.edu.cn
Wazir Ali
SMILE Lab, School of Computer Science and Engineering,
Clustering short text streams is a challenging task due to its unique properties: infinite length, sparse data representation and cluster evolution. Existing approaches often exploit short text streams in a batch way. However, determine the optimal batch size is usually a difficult task since we have no prior knowledge when the topics evolve. In addition, traditional independent word representation in the graphical model tends to cause "term ambiguity" problem in short text clustering. Therefore, in this paper, we propose an Online Semantic-enhanced Dirichlet Model for short text stream clustering, called OSDM, which integrates the word-occurrence semantic information (i.e., context) into a new graphical model and clusters for each arriving short text automatically in an online way. Extensive results have demonstrated that OSDM gives better performance compared to many state-of-the-art algorithms on both synthetic and real-world data sets.
A massive amount of short text data is constantly generated with online social platforms such as microblogs, Twitter and Facebook. Clustering of such short text streams has thus gained increasing attention in recent years due to many real-world applications like event tracking, hot topic detection, and news recommendation (Hadifar et al., 2019). However, due to the unique properties of short text streams such as infinite length, evolving patterns and sparse data representation, short text stream clustering is still a big challenge (Aggarwal et al., 2003; Mahdiraji, 2009).
During the past decade, many approaches have been proposed to address the text stream clustering problem from different points of view, and each method comes with specific advantages and drawbacks. Initially, traditional clustering algorithms for static data were enhanced and transformed for text streams (Zhong, 2005). Very soon, they are replaced by model-based algorithms such as LDA (Blei et al., 2003), DTM (Blei and Lafferty, 2006), TDPM (Ahmed and Xing, 2008), GSDMM(Yin and Wang, 2016b), DPMFP (Huang et al., 2013), TM-LDA (Wang et al., 2012), NPMM (Chen et al., 2019) and MStream (Yin et al., 2018), to mention a few. However, for most established approaches, they often work in a batch way, and assume the instances within a batch are interchangeable. This assumption usually cannot hold for topic-evolving text data corpus. Determining an optimal batch size is also a non-trivial task for different text streams (Howard and Ruder, 2018).
Additionally, unlike long text documents, short text clustering further suffers from the lack of supportive term occurrence to capture semantics (Gong et al., 2018). For most existing short text clustering algorithms like Sumblr (Shou et al., 2013), DCT (Liang et al., 2016) and MStreamF (Yin et al., 2018), exploiting independent word representation in their cluster models tends to cause ambiguity. Let us show the following four tweets, for example:
T1: "A regular intake of an Apple can improve your health and muscle stamina."
T1: "A glass of fresh apple juice is recommended for breakfast."
T2: "New Apple Watch can monitor your health."
*Corresponding author: Junming Shao
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 766-776 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
T2: "Apple will launch new smartphone iPhoneX this December."
Tweets of these two topics share few common terms, i.e., 'health' or 'apple'. It creates an ambiguity if the model deals with only single term representation to calculate the similarity. However, the co-occurring terms representation (i.e., context) helps a model to identify the topic<sup>1</sup> correctly.
To solve these aforementioned issues, we propose an online semantic-enhanced dirichlet model for short text stream clustering. Compared to existing approaches, it has following advantages. (1) It allows processing each arriving short text in an online way. The online model is not only free of determining the optimal batch size, but also lends itself to handling large-scale data streams efficiently; (2) To the best of our knowledge, it is the first work to integrate semantic information for model-based online clustering, which is able to handle "term ambiguity" problem effectively and finally support high-quality clustering; (3) Equipped with Poly Urn Scheme, the number of clusters (topics) are determined automatically in our cluster model.
During the past decade, many text stream clustering algorithms have been proposed. Here, due to the space limitation, we only report some model-based approaches which are highly related to our work. For more details, please refer to comprehensive surveys, e.g., (Mahdiraji, 2009; Silva et al., 2013; Nguyen et al., 2015; Aggarwal, 2018).
The early classical attempt for text clustering is Latent Dirichlet Allocation (LDA) (Blei et al., 2003). However, it cannot handle the temporal data for text streams. For this purpose, many LDA variants have been proposed to consider the text streams such as dynamic topic model (DTM) (Blei and Lafferty, 2006), dynamic mixture model (DMM) (Wei et al., 2007), temporal LDA (T-LDA) (Wang et al., 2012), streaming LDA (S-LDA) (Amoualian et al., 2016), and dirichlet mixture model with feature partition (DPMFP) (Zhao et al., 2016). These models assume that each document contains rich content, and thus they are not suitable for dealing with the short text streams. Later, Dirichlet multinomial mixture model-based dynamic clustering topic (DCT) model was designed to deal with short text streams by assigning each
document with single topic (Liang et al., 2016). Very soon, GSDMM was proposed to extend DMM with collapsed gibbs sampling to infer the number of clusters (Yin and Wang, 2014). However, most of these models did not investigate the evolving topics (clusters) in text streams where the number of topics usually evolves over time.
To automatically detecting the number of clusters, (Ahmed and Xing, 2008) proposed a temporal dirichlet process mixture model (TDMP). It divides the text stream into many chunks (batches), and assumes that the documents inside each batch are interchangeable. Later, GSDPMM was proposed with collapsed gibbs sampling to infer the number of clusters in each batch. In contrast to LDA, GSDPMM not only converges faster but also dynamically assigns the number of clusters over time (Yin and Wang, 2016a). However, both TDMP and GSDPMM models do not examine the evolving topics, and, these models process the text stream for multiple times. Thereafter, MStreamF (Yin et al., 2018) was thus proposed by incorporating a forgetting mechanism to cope with cluster evolution, and allows processing each batch only one time. The NPMM model (Chen et al., 2019) was recently introduced by using the word-embeddings to eliminate a cluster generating parameter of the model.
In summary, for most existing approaches, they usually work in a batch way. However, determining optimal batch sizes for different text streams is usually a difficult task. More importantly, due to the intrinsic sparse data representation of short-text data, the semantics, is little investigated in established approaches. Actually, they need to be carefully considered to decrease the term ambiguity in short text clustering.
Here, the problem statement is first given, followed with a brief introduction about dirichlet process and Poly Urn scheme.
Formally, a text stream is continuous arrival of text documents over time:  $S_{t} = \{d_{t}\}_{t = 1}^{\infty}$ . Where  $d_{t}$  denotes a document arrived at time  $t$ . Each document contains specific words  $d_{t} = \{w_{1}, w_{2}, \ldots, w_{n}\}$  and may have different length. The key objective of the clustering task is to group similar documents into clusters:  $Z = \{z_{t}\}_{t = 1}^{\infty}$ , and each clus
<sup>1</sup>Topic and cluster will be interchangeably used in this paper
ter  $z_{t}$  contains documents represented as  $z_{t} = \{d_{1}^{z_{t}}, d_{2}^{z_{t}}, \ldots, d_{n}^{z_{t}}\}$ . For short text clustering, each document is the member of only one topic, so  $z_{i} \cap z_{j} = \phi$ , where  $i \neq j$ .
3.2 Dirichlet Process
Dirichlet Process (DP) is a non-parametric stochastic processes to model the data (Teh et al., 2006). It is the process to draw a sample from (base) distribution, where each sample itself is a distribution, denoted as  $\mathcal{N} \sim \mathrm{DP}(\alpha, \mathcal{N}_0)$ . Here,  $\mathcal{N}$  is the drawn sample from the base distribution  $\mathcal{N}_0$ . The drawing procedure of a sample from the distribution is controlled by a concentration parameter  $\alpha$ .
3.3 Poly Urn Scheme (PUS)
The procedure to draw the sequential samples  $\mathcal{N}_1, \mathcal{N}_2 \ldots$  from a distribution is described by the poly urn scheme (Blackwell et al., 1973). It can be summarized as:
$$
\mathcal {N} _ {n} | \mathcal {N} _ {1: n - 1} \sim \frac {\alpha}{\alpha + n - 1} + \frac {\sum_ {k = 1} ^ {n - 1} \delta \left(\mathcal {N} _ {n} - \mathcal {N} _ {k}\right)}{\alpha + n - 1}
$$
Here,  $\delta(x) = 1$  if  $x = 0$  and  $\delta(x) = 0$  otherwise. Initially, the urn is empty, so we draw a color from the base distribution i.e.  $\mathcal{N}_1 \sim \mathcal{N}_0$ , and put a ball of drawn color into the urn. In the next turn, either we draw a color from the distribution which is already drawn with probability of  $\frac{n-1}{\alpha+n-1}$ , or draw a new color with probability of  $\frac{\alpha \mathcal{N}_0}{\alpha+n-1}$ . Since, drawing samples from distribution is repeated, so the same color may appear more than once. This defines that we have  $K$  number of distinct colors and  $n$  number of draws. This condition is defined by a well-known process called Chinese restaurant process (CRP) (Ferguson and Thomas S Ferguson, 1973). In CRP, we suppose that there are infinite number of tables in a restaurant, and each table surrounds infinite number of empty chairs. The first customer sits on first table, and later on the next customer either chooses to sit on any occupied table with probability of  $\frac{n_k}{\alpha+n-1}$  or chooses an empty table with probability of  $\frac{\alpha}{\alpha+n-1}$ . Here,  $n_k$  is number of customers sitting on a specific table. A new customer is tend to be attracted towards a highly crowded table. This phenomenon is one part of our equation to understand creation of clusters over time. The CRP represents the draws from distribution  $\mathcal{G}$ , while the stick-breaking process shows
the property of  $\mathcal{G}$  explicitly:
$$
\mathcal {G} (\mathcal {N}) = \sum_ {k = 1} ^ {\infty} \theta_ {k} \delta \left(\mathcal {N} - \mathcal {N} _ {k}\right), \quad \mathcal {N} _ {k} \sim \mathcal {N} _ {0} \tag {1}
$$
The mixture weights  $\theta = \{\theta_{k}\}_{k = 1}^{\infty}$  can be formalized by  $\theta \sim GEM(\gamma)$  (Neal, 2000). We exploit Equation (1) for the generative process of the Dirichlet process multinomial mixture model (DPMM) as follows.
$$
z _ {d} | \theta \sim \operatorname {M u l t} (\theta) d = 1, \dots , \infty
$$
$$
\mathcal {N} _ {k} | \beta \sim \operatorname {D i r} (\beta) \quad k = 1, \dots , \infty
$$
$$
d \left| z _ {d}, \{\mathcal {N} _ {k} \} _ {k = 1} ^ {\infty} \right. \sim p \left(d | \mathcal {N} _ {z _ {d}}\right)
$$
Here,  $z_{d}$  is the assigned documents to the cluster, which are multinomial distributed. The probability of document  $d$  generated by topic  $z$  is summarized as:
$$
p \left(d \mid \mathcal {N} _ {z}\right) = \prod_ {w \in d} \operatorname {M u l t} \left(w \mid \mathcal {N} _ {z}\right) \tag {2}
$$
Here, the naive Bayes assumption is considered where words in a document are independently generated by the topic. Whereas, the sequential draw of the sample can be derived by following the CRP. It is also assumed that the position of words in a document is not considered while calculating the probability.
4 Proposed Approach
This section gives a brief discussion about the representation and formulation of the proposed algorithm.
4.1 Model Representation
We build our model upon the DPMM (Yin and Wang, 2016a), which is an extension of the DMM model to deal with evolving clusters. We call our model as OSDM (Online Semantic-enhanced Dirichlet Model), aiming at incorporating the semantic information and cluster evolution simultaneously for short text stream clustering in an online way. The graphical model of OSDM is given in Figure 1a.
We show two major differences in our model to highlight the novelty. First, for word-topic distribution, we embed semantic information by capturing the ratio of word co-occurrence. Thereby, independent word generating process and word co-occurrence weight are well considered in topic generation. Secondly, our model works instance
(a) OSDM
(b) MStreamF
Figure 1: The graphical representation of OSDM and MStream. Here MStream works in a batch way while OSDM works in an online way.
by instance fashion to cluster the documents, instead of batch by batch. For comparison, Figure 1b further show the MStreamF (Yin et al., 2018) model. At initial stage before clustering documents of a batch, MStreamF update vocabulary set (active terms) from all the documents in a batch, then it starts the clustering each document of the batch. However, OSDM does not consider fixed number of documents to create vocabulary set, instead it incrementally updates with each arriving document.
4.2 Model Formulation
Defining the relationship between documents and clusters is the most crucial task while dealing with the text stream clustering problem. The threshold-based methodology (Nguyen et al., 2015) adapts similarity measures to define the homogeneity threshold between a cluster and a document. If the dissimilarity between the exiting clusters and a new arriving document is above the threshold, then a new cluster is created. However, due to the dynamic nature of the stream, it is very hard to define the similarity threshold manually.
In contrast, we assume that documents are generated by DPMM (see Section 3). Most recent algorithm MStreamF improved DPMM to cluster short text documents in the stream. As a further study, we integrate the semantic component in DPMM model. Additionally, we integrate term importance on the basis of cluster frequency. The derived equation for calculating the probability of a document  $d$  choosing existing cluster  $z$  is given in Equation
$$
\begin{array}{l} p \left(z _ {d} = z | \vec {z}, \vec {d}, \alpha , \beta\right) = \left(\frac {m _ {z}}{D - 1 + \alpha D}\right). \\ \left(\frac {\prod_ {w \in d} \prod_ {j = 1} ^ {N _ {d} ^ {w}} \left(n _ {z} ^ {w} \cdot l C F _ {w}\right) + \beta + j - 1}{\prod_ {i = 1} ^ {N _ {d}} n _ {z} + V \beta + i - 1}\right). \\ \left(1 + \sum_ {w _ {i} \in d \wedge w _ {j} \in d} c w _ {i j}\right) \tag {3} \\ \end{array}
$$
The first term of this Equation  $\left(\frac{m_z}{D - 1 + \alpha D}\right)$  represents completeness of the cluster. Here,  $m_{z}$  is the number of documents contained by the cluster  $z$  and  $D$  is the number of current documents in active clusters. Whereas,  $\alpha$  is the concentration parameter of the model. The middle term of the equation based on multinomial distribution (see Equation (2)) with psuedo weight of words  $\beta$  defines the homogeneity between a cluster and a document.  $N_{d}$  and  $N_{d}^{w}$  represents total number of words and term frequency of word  $w$  in document  $d$ , respectively. The symbol  $n_z^w$  is the term frequency of the word  $w$  in the cluster  $z$ . The current vocabulary size of the model is represented by  $V$ .  $n_z$  is the number of words in the cluster  $z$ .  $ICF_w$  calculates the term importance over the active clusters in the model, which is defined as follows.
$$
I C F (w \in d) = \log \left(\frac {| Z |}{| w \epsilon Z |}\right) \tag {4}
$$
Here,  $|Z|$  represents the number of active clusters in the model. The denominator part of Equation (4) is the number of those cluster which contains the word  $w$ . The term  $\left(1 + \sum_{w_i \in d \land w_j \in d} cw_{ij}\right)$  defines the semantic weight of term co-occurrence between the cluster and a document. Formally, we define a value of an entry  $cw_{ij}$  in the co-occurrence matrix as follows.
$$
c w _ {i j} = \frac {\sum_ {d ^ {\prime} \subseteq z} n _ {d ^ {\prime}} ^ {w _ {i}}}{\sum_ {d ^ {\prime} \subseteq z} n _ {d ^ {\prime}} ^ {w _ {i}} + \sum_ {d ^ {\prime} \subseteq z} n _ {d ^ {\prime}} ^ {w _ {j}}} \quad \left(w _ {i}, w _ {j}\right) \in d ^ {\prime} \tag {5}
$$
Here,  $n_z^{d'}$  is frequency count of word  $w_i$  in document  $d'$ . The ratio between  $w_i$  and  $w_j$  must satisfy the property  $cw_{ij} + cw_{ji} = 1$ . We calculate the term co-occurrence weight of those terms which are
2Active clusters refer to those clusters which are not yet deleted from the model.
common in the cluster  $z$  and document  $d$ . Term co-occurrence matrix is constructed where two terms are co-occurred in a single document. Therefore, if the size of cluster feature set (discussed in Section 4.3) is  $|V_z|$ , then it is not necessary that the co-occurrence matrix would be  $|V_z| \times |V_z|$ .
So far, we have defined the probability of a document choosing existing cluster, then we have to define the probability for a document to creating a new cluster. By following the DPMM for infinite number of clusters, which transform  $\theta \sim GEM(\gamma)$  into  $\theta \sim GEM(\alpha D)$ , because the hyper-parameter for the mixture model should be dynamically change over time. Therefore, the probability of creating a new cluster is as follows.
$$
\begin{array}{l} p \left(z _ {d} = z | \vec {z} _ {\neg d}, \vec {d}, \alpha , \beta\right) = \left(\frac {\alpha D}{D - 1 + \alpha D}\right) \\ \cdot \left(\frac {\prod_ {w \in d} \prod_ {j = 1} ^ {N _ {d} ^ {w}} \beta + j - 1}{\prod_ {i = 1} ^ {N _ {d}} V \beta + i - 1}\right) \tag {6} \\ \end{array}
$$
Here, the pseudo number of clusters related documents in the model is represented as  $\alpha D$ , and  $\beta$  is the pseudo term frequency of each word (exist in document) of the new cluster.
4.3 The cluster feature (CF) set
The similarity-based text clustering approaches usually follow vector space model (VSM) to represent the cluster feature space (Din and Shao, 2020). However, a topic needs to be represented as the subspace of global feature space. Here, we use a micro-cluster feature set to represent each cluster. Namely, a cluster is represented as the summary statistics of a set of words of related documents. In our model, a cluster feature (CF) set is defined as a 6-tuple  $\{m_z, n_z^w, cw_z, len_z, l_z, u_z\}$ , where  $m_z$  is the number of documents in the cluster  $z$ ,  $n_z^w$  is the number of frequency of the word  $w$  in the cluster,  $cw_z$  is the word to word co-occurrence matrix,  $len_z$  is the number of words in the cluster  $z$  which is sum of all frequencies of words,  $l_z$  is the cluster weight, and  $u_z$  is the last updated time stamp.
The desirable addition property of cluster feature allows updating each micro-cluster in an online way.
Definition 1: A document  $d$  can be added to a cluster  $z$  by using the addition property.
$$
\begin{array}{l} m _ {z} = m _ {z} + 1 \\ n _ {z} ^ {w} = n _ {z} ^ {w} + N _ {d} ^ {w} \quad \forall w \in d \\ c w _ {z} = c w _ {z} \cup c w _ {d} \\ \end{array}
$$
Algorithm 1: OSDM
Input:  $S_{t}:\{d_{t}\}_{t = 1}^{\infty}$ $\alpha$  : concentration parameter,  $\beta$  : pseudo weight of term in cluster,  $\lambda$  : decay factor
Output: Cluster assignments  $z_{d}$
1  $K = \phi$
2 while  $d_t$  in  $S_t$  do
3  $t = t + 1$
4  $K = \text{removeOld} Z_{i}(K)$
5  $K =$  reduceClusterWeight  $(\lambda ,K)$
6 | foreach  $z_{i} \in K$  do
7  $P_{Z_i} = \text{prob}(z_i, d_t)$  using Eq. (3)
8 end
9  $i = \arg \max_{i}(P_{Z_i})$
10  $P_{Z_n} =$  calculate the probability of new cluster using Eq. (6)
11 if  $P_{Z_i} < P_{Z_n}$  then
12  $m_{z_n} = 1$
13
14  $cw_{z_n} = cw_{d_t}$
15 | lenzn = lendt
16  $l_{z_n} = 1, u_{z_n} = t$
17  $K = K\cup z_{n}$
18 else
19  $m_{z_i} = m_{z_i} + 1$
20  $n_{z_i}^w = n_{z_i}^w +N_{dt}^w$
$cw_{z_i} = cw_{z_i}\cup cw_{d_t}$
$len_{z_i} = len_{z_i} + len_{d_t}$
$l_{z_i} = 1, u_{z_i} = t$
24 end
25 end
$$
l e n _ {z} = l e n _ {z} + l e n _ {d}
$$
Here,  $cw_{d}$  is word to word co-occurrence of the document, and  $len_{d}$  represents the number of total words in the document. The complexity of updating a cluster by adding a document is  $O(\mathcal{L})$ , where  $\mathcal{L}$  is the average length of the document. This property is useful to update evolving micro-clusters in the text stream clustering procedure.
4.4 OSDM Algorithm
We propose a semantic-enhanced non-parametric dirichlet model to cluster the short text streams in an online way, called OSDM. The proposed algorithm allows processing each instance incrementally and updates the model accordingly.
The procedure of OSDM is given in Algorithm 1. Initially, it creates a new cluster for the first doc
ument and the document is assigned to the newly created CF set. Afterward, each arriving document in the stream either choose an existing cluster or generate a new cluster. The corresponding probability for choosing either of an existing cluster or a new cluster is computed using Equation (6) and (3), respectively. The CF vector with the highest probability is updated using the addition property.
To deal with the cluster evolution (i.e., evolving topics) in text streams, many existing approaches often delete the old clusters by using some of the forgetting mechanisms (e.g., decay rate) (Zhong, 2005; Aggarwal and Yu, 2010; Islam et al., 2019). Instead of deleting old clusters, MStreamF (Yin et al., 2018) deletes old batches. In this study, we investigate the importance of each micro-cluster to handle the cluster evolution problem. Specifically, the importance of each micro-cluster is decreased over time if it is not updated.  $l_{z}$  in CF stores weight of each cluster. If the weight is approximately equals to zero, then the cluster is removed from the model, i.e., it cannot capture recent topics in the text stream. For this purpose, we applied the exponential decay function,  $l_{z} = l_{z} \times 2^{-\lambda \times (\Delta t)}$ . Here,  $\Delta t$  is the elapsed time from the last update, and  $\lambda$  is the decay rate. The decay rate must be adjusted depending upon the applications at hand. The initial value of  $l_{z}$  (See Line 16 of Algorithm 1) is set to 1. Afterward, the importance of micro-cluster is exponentially decreases over time. We can also store the deleted clusters in a permanent disk for offline analysis.
Complexity Analysis. The OSDM algorithm always maintains the average  $\bar{K}$  number of current topics (CF sets). Every CF set store average  $\bar{V}$  number of words in  $n_z^w$  and at most  $|\bar{V}_z| \times |\bar{V}_z|$  in  $cw_{z}$ . Thus the space complexity of OSDM is  $O(\bar{K} (\bar{V} +\bar{V}^{2}) + VD)$ , where  $V$  is the size of active vocabulary and  $D$  is the number of active documents. On the other side, OSDM calculates the probability of arriving document with each cluster (see Line 6 of Algorithm 1). Therefore, the time complexity of OSDM is  $O(\bar{K} (\mathcal{L}\bar{V}))$ , where  $\mathcal{L}$  is the average size of arriving document.
5.1 Datasets and evaluation metrics
To evaluate the performance of the proposed algorithm, we conduct experiments on three real and two synthetic datasets. These datasets were also used in (Yin and Wang, 2016a; Liang et al., 2016;
Qiang et al., 2018; Yin et al., 2018; Jia et al., 2018; Chen et al., 2019) to evaluate short text clustering models. In the preprocessing step, we removed stop words, converted all text into lowercase, and stemming. The description of the datasets is as follows.