# MSA K-BERT: A Method for Medical Text Intent Classification
## 1 Introduction  
Medical text intent [1] refers to identifying the intended purpose of a paragraph from medical field texts, including patient questions, doctor responses, and medical records in natural language processing (NLP), providing an understanding basis for medical NLP systems. Due to the fact that medical texts contain many obscure medical professional terms and usually do not follow natural grammar [2], medical text intent classification is confronted with problems such as poor model generalization ability and interpretability, low quality and quantity of data annotation, and the need for updates in the construction of the classification system. Medical text intent classification methods have evolved from rule-based template matching to data-driven approaches. Early rule-based methods [3] are being gradually replaced by machine learning methods [4, 5, 6]. Modern models no longer rely on manually defined rule templates; instead, they automatically learn semantic patterns and decision boundaries from labeled data through algorithms. In recent years, deep learning methods [7, 8] become mainstream due to their ability to perform automatic feature extraction. In deep learning approaches, convolutional neural networks (CNNs) [9] and Bidirectional Encoder Representations from Transformers (BERTs) [10] currently represent state-of-the-art methods for many medical text intent classification tasks. Single CNN models [11] such as Text attention CNN (Text-CNN) [12; 13] and Dynamic CNN (DCNN) [14] typically rely on convolutional operations to extract local features, though they may exhibit limitations in modeling long-range dependencies and sequence positional information. Hybrid CNN [15] architectures that integrate Recurrent Neural Network (RNN) [9] or Attention mechanisms [16], such as CNN-RNN [17] and CNN-Attention [18] models, effectively address these limitations. While these approaches demonstrate strong performance on biomedical datasets, including PubMed 200 K RCT [19] and BioASQ [20], they continue to present several research challenges regarding the optimal configuration of convolution kernels, handling of imbalanced data distributions, and effective incorporation of external knowledge and domain expertise. BERT-based models for medical text intent classification [21; 22] have emerged as a new mainstream approach due to their pre-trained semantic representation capabilities. Single BERT models [22] (e.g., BERT-base [21], BERT-large [23]) typically achieve a strong baseline performance through direct fine-tuning, while hybrid BERT [24] architectures (e.g., BERT-CNN [25], BERT-BiLSTM [26]) further combine the advantages of local feature extraction and contextual modeling. In tasks such as the classification of randomized controlled trial titles, abstracts, biomedical question answering, and medical literature retrieval, BERT-based models generally demonstrate superior performance compared to traditional deep learning models. However, they still face limitations in optimizing pre-training strategies and processing multimodal data. Particularly when incorporating external knowledge, challenges like heterogeneous embedding space (HES) [27] and knowledge noise (KN) [28] remain critical bottlenecks affecting model performance. Moreover, although Large Language Models (LLMs) [29] like GPT [30] and Large Language Model Meta AI (LLMA) [29] achieve impressive results in tackling various complex tasks, they perform poorly in domain-specific classification tasks due to limitations such as resource constraints and explainability. [PERSON] et al. [31] study the performance of various models on medical intent classification tasks. The results show that models specifically pre-trained for the medical domain, such as RoBERTa [32] and PubMedBERT [31], significantly outperform general-purpose large language models like ChatGPT [30] and LLaMA-2 [33] across multiple datasets. Specifically, on the CMID dataset [1], RoBERTa achieves an accuracy of 72.88%, while ChatGPT achieves only 42.36%.