# Analysis of Short Texts Using Intelligent Clustering Methods
## 2 Materials and Methods  
TF-IDF achieves high accuracy in keyword detection but loses grammatical and contextual information. LDA models topics but ignores word order and context. BERT captures contextual dependencies but does not provide explicit topic features. Therefore, using these methods in isolation does not ensure reliable clustering of short, semantically rich texts. To address these limitations, this study proposes an end-to-end approach to text data analysis and clustering that integrates two advanced natural language processing technologies: LDA and BERT. This hybrid method, referred to as dependent embedding, is specifically designed to combine the strengths of each model while compensating for their individual weaknesses. In addition, TF-IDF is incorporated to improve the estimation of word importance in texts. The LDA method (Figure 1), as a statistical approach to topic modeling, enables the identification of common topics in large text collections. However, its main disadvantage lies in its inability to capture word order and meaning, which significantly limits its applicability for tasks requiring semantic understanding. In response to this limitation, BERT was introduced, a deep learning model based on the transformer architecture, which can analyze each word in all other words in a sentence, thereby capturing the contextual nuances of language. The analysis begins with data preprocessing, including removing text noise, such as special characters and stop words, and tokenizing and lemmatizing words. LDA is then applied to identify broad topic clusters, which helps identify common themes in a collection of texts. BERT is then used to create vectors of each word, allowing for a deeper understanding of the semantic relationships and nuances present in the texts (Figure 2). This study uses the TF + IDF method to further improve the quality of analysis and clustering. This method estimates the importance of each word in a single document based on its frequency of occurrence compared to its overall frequency across all documents in the corpus. This approach to weighting terms makes it possible to identify the most significant keywords for specific texts. It provides a more accurate and informative representation of texts for subsequent stages of clustering (Figure 3). The metrics used to evaluate clustering quality include the following: silhouette--indicates how objects are separated from each other during clustering (the higher, the better); calinski_harabasz index--evaluates intra-cluster and inter-cluster variation (the higher, the better); davies_bouldin--an averaged measure of cluster \"similarity\" (the lower, the better); adjusted_rand (ARI)--compares predicted clusters with ground truth labels (the higher, the better); homogeneity--shows how well each cluster contains objects from only one accurate category; completeness--reflects how well objects of the same category fall into one cluster; v_measure--a harmonic mean between homogeneity and completeness (the higher, the better). The parameter \(\gamma\) controls the relative weight of \"topic\" features (LDA vector) compared to \"contextual\" features (BERT vector), where increasing \(\gamma\) increases the contribution of topic-based features (LDA) when forming the feature space (1):  
\[vec_{ldabert}=np.c[vec_{lda}*self.gamma,vec_{bert}] \tag{1}\]  
Figure 1: The architecture of the LDA method. Figure 2: BERT method architecture. where \(np.c\)--concatenates two arrays: (LDA * \(\gamma\)) and BERT. We selected \(\gamma\) = 15 based on a series of preliminary experiments (values ranging from 1 to 20), as gamma = 15 achieved the best balance between topic-based and semantic information (Table 1). The hybrid LDA + BERT + AE method is an advanced approach to text data analysis, combining LDA for topic modeling, deep learning with BERT, and dimensionality reduction using autoencoders (AEs).