# Topic Modeling for Faster Literature Screening Using Transformer-Based Embeddings
## 1 Introduction  
The advancement of technology and the internet have ushered in a new era of information sharing that is significantly transforming how research is conducted at every stage [1]. Scholars and researchers can now effortlessly publish a vast array of data, reviews, and opinions across globally accessible platforms through new publishing models that simplify and accelerate the sharing of data and knowledge. However, publishing not only serves to disseminate knowledge for furthering research but also involves complex dynamics related to career progression and securing grant funding [2, 3]. As a result, an overwhelming number of publications covering diverse topics are published every day, making it challenging to efficiently identify relevant papers amidst a sea of sometimes only tangentially related literature [4]. The challenge becomes particularly critical when conducting systematic reviews. Systematic reviews are rigorous and comprehensive examinations of the existing literature to answer specific research questions [5] that rely on experimental--and sometimes observational--studies, with the purpose of gathering all the existing evidence on a given medical condition and, usually, its therapy. To conduct a successful systematic review,researchers must consider a wide range of sources and databases to ensure comprehensive coverage of the relevant literature [5; 6] and to make sure to identify all the pertinent data, minimizing the risk of missing relevant evidence. For this purpose, researchers usually rely on established library repositories to search for scientific articles [7]. These databases, such as Medline, are commonly searched using specific keywords that may appear in the article, e.g., in its title or abstract. Though this approach is fast and robust, it may fail to capture relevant articles if they use different wordings or synonyms [8]. It has long been recognized that multiple searches with complex syntax, which requires advanced query construction skills, are often needed to narrow down the search space [9]. Moreover, as the lexicon may be ambiguous, this approach usually yields large numbers of publications that do not match the exact focus of the query--or are even off topic--and forces researchers to manually sift through the query results to handpick the papers they need [10]. Considerable research has previously focused on developing and testing search filters designed to refine results by targeting specific criteria [11; 12; 13; 14; 15]. Automation has great potential to improve literature searches and expedite systematic reviews [16]. Recent advancements in natural language processing (NLP) and machine learning have demonstrated the possibility to automate or assist several tasks within the systematic review process [17; 18; 19]. Innovations in this area have led to the development of software like Abstract, ASReviews, EPPI-reviewer, and RobotSearch, which utilizes convolutional neural network architectures to identify RCTs [20; 21]. Large language models are promising tools to automate some aspects of systematic reviews by enhancing literature retrieval through semantic understanding and the contextual analysis of search terms [22; 23]. An essential component for achieving semantic understanding is embeddings--numerical representations that encode word or even sentence meaning--which are key in NLP for capturing complex relationships between words and sentences using special architectures known as transformers [24]. Embeddings based on transformer architectures have proven very effective in several NLP tasks, and many efforts have been devoted to developing better embeddings for specific tasks, also in the biomedical field, including topic modeling [25]. Topic modeling involves identifying the main theme of unlabeled documents [26]. This approach can be valuable in understanding complex scientific literature corpora by automatically organizing and categorizing large sets of publications based on their topics [27].