## 3 Methodology
### Enhancing BERTopic-Based Framework for Dynamic Topic Modeling
#### 3.2.1 Embedding Method Benchmark for Topic Extraction  
Text embedding represents each unique word in a corpus as an N-dimensional vector. In essence, it maps words into an N-dimensional vector space, from which valuable insights, such as the semantic similarity between different words, can be deduced. When aiming to identify an effective and efficient text embedding method for unraveling dynamic technological topic evolution in 6G communication, we evaluated the performance of the aforementioned embedding approaches in terms of their ability to capture content semantics and extract technological topics. To guarantee a fair comparison during the model selection phase, we executed the document clustering tasks using the same clustering algorithm. Term Frequency-Inverse Document Frequency (TF-IDF) [54] is an initial textual embedding technique. Essentially, it is one-dimensional text embedding, which can effectively represent the importance of words in a text within the context of a document collection. However, TF-IDF has several limitations when capturing the dynamic nature of technological topics in 6G communication. Firstly, it is a static approach that only considers the frequency of words in documents without considering the semantic context in which they are used. In the rapidly evolving field of 6G, words may have different meanings depending on the specific technological context, and TF-IDF fails to capture these nuances. Furthermore, TF-IDF does not account for the emergence of new words and concepts that are characteristic of dynamic technological evolution. As 6G technology develops, new terminologies are constantly being introduced to describe novel features and technologies. TF-IDF, relying on historical document frequencies, may not be able to assign appropriate weights to these new words, potentially missing important aspects of the evolving technological topics. In this study, we also investigated static embedding techniques, such as Doc2 Vec (Word2 Vec), which are based on predicting word probabilities to map words into low-dimensional dense vectors. This can be obtained through two neural network methods: Skip Gram and CBOW. In Skip Gram, the target word is input to predict neighboring words;  
Figure 3: This figure is an example of our LLM-based summarization process for enhancing the input data in the pre-processing module. in CBOW, neighboring words are input to predict the target word. This way, Word2 Vec can capture semantic relationships between words, placing related words closer in its vector space. However, Word2 Vec has difficulty fully grasping the context-sensitive and specialized meanings of some 6G-specific terms. In addition, its embedding quality is highly dependent on the training data. In case the fast-evolving field does not cover all aspects of 6G technology, the embedding may not accurately represent word relationships. Therefore, we further introduced contextual embedding techniques. They leverage pre-trained transformer-based LLMs, such as Data2 Vec [55] and SBERT [56], to effectively capture the semantic and contextual information of words in a document. The attention mechanism in transformers allows for a more nuanced understanding of the relationships between words, enabling it to handle the complex semantic landscape of 6G technology. Furthermore, we considered the PatentSBERTa [33] in this study, which is specially trained on patent data and has never been studied in a DTM before. We conducted comparative experiments to evaluate these contextual embedding techniques and obtain a suitable embedding model for our DTM framework. In the context of a document corpus \(D\), each document (title and abstract) \(d\) is assigned a vector representation \(y\). More precisely, the document embedding \(y\) for a given document \(d\) is obtained through a mapping function \(F\), denoted as \(F:d\mapsto y\in\mathbb{R}^{z}\). This mapping effectively captures both the contextual and semantic information from the corpus.