# Short Text Clustering Algorithms, Application and Challenges: A Survey
## 3 Components of Short Text Clustering
### Global term weight  
It calculates the weight by collecting all training documents [78]. It tries to grant a discrimination value to each term and emphasize discriminatory terms. For example, the most popular and notable metric global term weighting schemes shown in Table 1, such as _idf_, _bidf_, and _pidf_, are unsupervised because they do not use the category label information from training documents. In _idf_, the main idea of the inverse document frequency is to provide high weights for rare terms and low values for standard terms. This scheme is calculated using the logarithmic ratio of the number of documents in a collection to the number of documents containing a specific term. The versions of _bidf_ and _pidf_ are the other two approaches to _idf_. The premise behind _idf_, _bidf_ and _pidf_ is that a phrase that appears less frequently in documents is more discriminatory. This strategy works well in IR, but it is inappropriate for text categorization and text clustering because these tasks are designed to distinguish between categories, not documents [78]. Table 1 shows that \(a\) and \(b\) represent the number of training documents in group one, including the terms \(t_{i}\) and \(c\); \(d\) represents the number of training documents in group two, including term \(t_{i}\). \(N\) represents the number of documents in the corpus, \(N=a+b+c+d\). Finally, text data are represented by considering the context or topic of text documents or segments. However, a document is characterized by its topic or the keywords that stand out the most. Consequently, several topics can be associated with a single document, and documents are clustered according to the number of topics they share. Latent Dirichlet Allocation (LDA)  
LDA is defined as generative probabilistic modelling for text data. LDA is one of the most widely used methods in topic modelling, and it was developed in 2003 by [73]. The fundamental concept is based on the texts represented as random mixtures from latent topics, where the distribution of the words characterizes a topic. The simplicity and effectiveness of LDA lead to its widespread use. LDA uses word probabilities to represent topics. The words with the highest probabilities in each topic usually give a good idea of what the topic is using word probabilities from LDA. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Term Weights** & \multicolumn{2}{c}{**Formulation**} & **Description** \\ \hline \multirow{7}{*}{Local weights} & _tf_ & _tf_ & Raw term frequency \\ \cline{2-4}  & _tp_ & \(\left\{\begin{matrix}1,\text{ if }tf>0\\ 0,\text{ otherwise}\end{matrix}\right.\) & Appearance of the term \\ \cline{2-4}  & _atf_ & \(k+(1-k)\frac{tf}{\text{max}_{t}(tf)}\) & \(\text{max}_{t}(tf)\) indicates the highest term frequency \\ \cline{2-4}  & _ltf_ & \(\log_{2}(1+t)\) & Logarithm of term frequency \\ \cline{2-4}  & _btf_ & \(\frac{(k_{1}+1)+tf}{k_{1}\left((1-b)+\frac{1+df}{\text{max}_{t}-u}\right)}+tf\) & \(\text{\begin{matrix}\text{\emph{aver}}_{\text{\emph{all}}}\text{ represents the average}\\ \text{number of terms found in}\end{matrix}\)} \\ \hline \multirow{7}{*}{Global weights} & _i df_ & \(\log_{2}\frac{N}{a+c}-1\) & Inverse document frequency \\ \cline{2-4}  & _pi df_ & \(\log_{2}\left(\frac{N}{a+c}-1\right)\) & Probabilistic _idf_ \\ \cline{1-1} \cline{2-4}  & _bi df_ & \(\log_{2}\frac{b+d+0.5}{a+c+0.5}\) & BM25 _idf_ \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Local and global term weighting schemes. LDA assumes that each text may represent a probability distribution across latent topics, with a shared Dirichlet prior across all texts. Each latent topic is represented as a probabilistic distribution from words in the LDA model, and the word distributions of topics share a common Dirichlet prior. Assuming a corpus \(D\) consisting of \(M\) documents, with document \(d\) having \(N_{d}\) words (\(d\in 1,\ldots,M\)), LDA models \(D\) using to the following generative process [73]:  
1.