# MSA K-BERT: A Method for Medical Text Intent Classification
## 3 Related Work  
### Text Classification  
Early text classification methods are primarily based on machine learning techniques, which focus on enabling computer systems to learn from data and continuously promote their performance without the need for explicit programming. This technology makes computers learn from experience and make informed decisions or predictions on new data. [PERSON] et al. [39] design a Least Squares Support Vector Machine (LS-SVM) for text classification of noisy document titles. The method achieves a classification accuracy exceeding 99% when evaluated on a corpus of 91,229 words from the Penrose Library catalog at the University of Denver. [PERSON] et al. [40] implement a hybrid enhancement method using Naive Bayes and SVM, which improves the average classification accuracy by 1.05% compared to the standard SVM and by 4.41% compared to the Bayesian classifier. To effectively extract complex features from unbalanced text data, [PERSON] et al. [41] propose an ensemble method based on Random Forest (RF), called ForesTexter. This model outperforms Random Forest in fitting imbalanced data, and its AUC score is 3.27% higher than the corresponding score of RF. Machine learning algorithms perform well when dealing with small-scale data [42], mainly because of their low computational resource requirements and fast model training. However, traditional machine learning methods have some limitations when dealing with large-scale, high-dimensional sparse data. Therefore, to solve the above-mentioned problems, the emergence of deep learning techniques [8; 43] has brought significant breakthroughs and innovations in the text classification field. The introduction and improvement of deep learning models have achieved significant performance enhancements in the text classification task. [PERSON] et al. [13], considering the significant impact of noisy background information on extracting real text, propose a new text attention convolutional neural network, namely Text-CNN, which emphasizes extracting text features from image components and distinguishing effective text regions. Compared with the traditional CNN model, this model has increased the average accuracy of text recognition and classification to 91%. To effectively handle the high dimensionality, sparsity, and complex semantics of text data, [PERSON] et al. [44] propose the AC-BILSTM model, which is a bidirectional long short-term memory network with an attention mechanism and convolutional layers. This model is evaluated on seven comprehensive labeled datasets. In comparison with the previous state-of-the-art text classification methods, this model demonstrates outstanding classification accuracy and robustness. In a multidimensional sentiment text classification analysis, [PERSON] et al. [45] propose a tree-structured CNN-LSTM model, composed of region-based CNN and LSTM, to predict sentiment information ratings. By comparing with other LSTM and CNN models, the results indicate that the Regional CNN-LSTM (Tree) model, which combines region division with tree depth structure information, exhibits outstanding predictive performance. The model achieves a Mean Squared Error (MSE) of 0.94, and the observed performance differences are statistically significant (\(p<0.05\)). It can be seen that neural network-based deep learning models [46; 47] have high accuracy and performance when dealing with large-scale or high-dimensional data. However, neural network models usually require substantial labeled data, computational resources, and a long training time when performing large-scale training tasks [48; 49]. In terms of training effectiveness, neural network models still lack significant generalization ability when applied to different domains or tasks. Over the past few years, pre-training models such as BERT [10] have been proposed and widely used as training strategies for deep learning models.