###### Abstract  
As an extension of the transformer architecture, the BERT model has introduced a new paradigm for natural language processing, achieving impressive results in various downstream tasks. However, high-performance BERT-based models--such as ELECTRA, ALBERT, and RoBERTa--suffer from limitations such as poor continuous learning capability and insufficient understanding of domain-specific documents. To address these issues, we propose the use of an attention mechanism to combine BERT-based models with neural topic models. Unlike traditional stochastic topic modeling, neural topic modeling employs artificial neural networks to learn topic representations. Furthermore, neural topic models can be integrated with other neural models and trained to identify latent variables in documents, thereby enabling BERT-based models to sufficiently comprehend the contexts of specific fields. We conducted experiments on three datasets--Movie Review Dataset (MRD), 20  
information contained within the documents. Historically, probabilistic topic models, such as latent Dirichlet allocation (LDA) [14], have been widely used for this task. However, topic models based on Bayesian probabilities have three main drawbacks [15]. First, as model complexity increases, so does the complexity of inference, making it difficult to automate the inference process. Furthermore, it is challenging to efficiently process large text sets because parallel computing technologies such as GPUs cannot be utilized. Finally, these models cannot be trained in conjunction with artificial neural networks, including deep neural networks.  
The neural topic model (NTM) has emerged to address these limitations. Unlike conventional topic modeling methods such as LDA, NTM uses artificial neural networks to discover latent topics in documents without relying on probabilistic approaches. This allows various frameworks to be applied to configure neural topic modeling.  
In this study, we propose a method to address the challenges of continuous learning and the limitations of pre-trained models, which often struggle to accurately comprehend domain-specific documents. Our approach simultaneously utilizes pre-trained models and NTMs to create a more effective solution. Instead of simply loading pre-trained models and applying them directly to the task, we incorporate a brief additional training step after loading the pre-trained models. This step occurs before executing the main task and is designed to refine the language model's capabilities through a straightforward fine-tuning process, eliminating the need for extensive pre-training. Our experimental results, obtained from three benchmark datasets--MRD, 20 Newsgroups, and YELP--demonstrate the enhanced performance of the proposed approach, validating our methodology.  
The structure of the paper is as follows: Section 2 explores traditional topic modeling and subsequent developments in neural topic modeling, providing a foundational understanding of the field. Section 3 focuses on the fundamental BERT-based models and outlines the architecture of our proposed model, highlighting its unique features and advantages. Section 4 details the datasets and hyperparameters utilized in our experiments, presents the experimental results, and includes additional analyses based on variations in specific hyperparameters. Finally, Section 5 offers a comprehensive summary of the experiments and discusses potential avenues for future research, emphasizing the significance of our findings and their implications for ongoing developments in this area.