# Topic Modeling for Faster Literature Screening Using Transformer-Based Embeddings
## 2 Materials and Methods
### Data Analysis  
The data were analyzed using Google Colab Pro notebook powered by Python 3.10.12 [37] and running on T4 GPUs [38], which provide the acceleration required to handle embeddings efficiently. The analysis of the publications was conducted on their titles, based on the assumption that titles are a summary of the content of a paper and are thus representative of their topic [39; 40]. The datasets did not need to undergo any preprocessing other than removing entries when titles were missing. Unlike previous publications [41], we did not deem it necessary to lowercase the titles, nor to remove stopwords, to rely on BERTopic's capability to produce contextual embeddings. Unlike bag-of-words approaches, there is a consensus that stopwords may actually improve sentence encoding by providing further context using transformer architectures [42]. Embeddings in NLP are dense vectors that represent the semantics of words in a multi-dimensional space [43]. Unlike older algorithms [44], bidirectional encoder representations from transformers (BERT) understands the context and creates unique word embeddings based on their usage in different contexts [24; 45]. BERTopic operates through several stages, including transformer embedding models, dimensionality reduction, clustering, and cluster tagging using cTF-IDF [28], which are summarized in Figure 1.  
The first step is the creation of the embeddings. To carry that out, we chose the [PERSON]'s 'all-mpnet-base-v2' model. Figure 1: Diagram illustrating the workflow used in the present work to model the topics in our datasets. Our initial dataset was in tabular form; titles were converted into embeddings, which were then reduced by UMAP. Reduced embeddings were clustered by HDBSCAN based on their similarity, and keyword descriptors were generated for every cluster by cTF-IDF. A large language model (LL) was then used to create convenient labels for the topic, converting the keywords into a sentence. The embeddings from this model are too large for efficient clustering and must therefore be reduced. As several dimensionality reduction algorithms are available, we used uniform manifold approximation and projection (UMAP), which has been shown to be very effective in preserving the topological structure of data [46]. We empirically decided to reduce all-mpnet-base-v2's 768-dimension embeddings to 5-dimension embeddings for subsequent processing and to 2-dimensions for visualization. The reduced embeddings were then clustered with hierarchical density-based spatial clustering of applications with noise (HDBSCAN) [47], and cTf-Idf was applied to extract topic keywords in each cluster. Unlike Tf-Idf [48], cTf-Idf adjusts the weight based on the term frequency within a cluster of documents rather than within an individual document [49]. More specifically, we decided to use the following set-up:  
* [noitemsep]  
labeling is efficient but its result is admittedly often obscure. Fortunately, BERTopic allows the integration of other algorithms to create better representations of topics, including large language models (LLM). An LLM is an A.I. algorithm that can generate human-like responses [52] and can create more comprehensive and representative descriptions of topics, i.e., better labels. We decided to use the freely available OpenHermes-2.5-Mistral large language model [53]. LLMs are typically--as their name suggests--very large and require vast computer resources, thus reduced versions have been elaborated [54] that are commonly referred to as quantized LLMs [55]. We opted for the OpenHermes-2.5-Mistral-7B-GGUF/openhermes-2.5-mistral-7b.Q4_K_M_gguf quantization, available for download on Huggingface.com (accessed on 3 March 2024). LLMs need a prompt from the users [56] to generate a response, and we set the following prompt:  
_\"\"_Q:  
I have a topic that contains the following documents:  
[DOCUMENTS]  
The topic is described by the following keywords: [KEYWORDS]'. Based on the above information, can you give a short label of the topic of at most 5 words? A:  
_\"