# MSA K-BERT: A Method for Medical Text Intent Classification
## 5 Results
### Comparative Experiment  
In this experiment, we evaluate seven text classification models, including TEXTCNN, BERT-CNN, ALBERT, BERT, ERNIE, K-BERT, and our proposed MSA K-BERT. These models are applied to a medical text classification task, with accuracy, recall, and F\({}_{1}\)-score adopted as evaluation metrics. All models are evaluated on the same dataset: IMCS-21. Table 3 provides a summary of the experimental results. From Table 3 and Figure 5, it can be observed that the MSA K-BERT model outperforms all other models in all evaluation indicators. Specifically, in Table 3, we perform five repeated experiments for both the baseline model and MSA K-BERT, calculating the mean and standard deviation of accuracy, recall, and F\({}_{1}\)-score. The results demonstrate that the observed average performance improvement of our method over the baseline model is not due to random factors but is statistically significant. \begin{table}
\begin{tabular}{c c c c} \hline
**Intent Category** & **Training Set** & **Validation Set** & **Test Set** \\ \hline Request Symptom & 2957 & 8871 & 3020 \\ Inform Symptom & 4348 & 13,001 & 4481 \\ Request Etiology & 289 & 917 & 298 \\ Inform Etiology & 527 & 1498 & 508 \\ Request Basic Information & 1010 & 3056 & 988 \\ Inform Basic Information & 1659 & 4787 & 1528 \\ Request Existing Exam & 1268 & 3986 & 1313 \\ and Treatment & 1938 & 5953 & 1987 \\ Request Drug & 1256 & 3325 & 1061 \\ Recommendation & & & \\ Inform Drug & 2366 & 6712 & 2118 \\ Recommendation & & & \\ Request Medical Advice & 425 & 1219 & 373 \\ Inform Medical Advice & 1320 & 3770 & 1157 \\ Request Precautions & 326 & 904 & 400 \\ Inform Precautions & 1624 & 4906 & 1752 \\ Diagnose & 746 & 2261 & 737 \\ Other & 11,208 & 33,363 & 11,214 \\ Total & 33,267 & 98,529 & 32,935 \\ \hline \end{tabular}
\end{table} 
Table 2: Number of sample items for the IMCS-21 medical dialog dataset. \begin{table}
\begin{tabular}{c c c c c} \hline
**Model** & **Parameters** & **Accuracy** & **Recall** & **F\({}_{1}\)** \\ \hline TEXTCNN & 2 M & 0.720 & 0.570 & 0.603 \\ BERT-CNN & 110 M & 0.750 & 0.676 & 0.674 \\ AlBERT & 11 M & 0.783 & 0.695 & 0.714 \\ BERT & 110 M & 0.788 & 0.706 & 0.717 \\ ERNIE & 193.7 M & 0.794 & 0.739 & 0.743 \\ K-BERT (baseline) & 340 M & 0.807 \(\pm\) 0.004 & 0.727 \(\pm\) 0.006 & 0.736 \(\pm\) 0.005 \\ MSA K-BERT & 310 M & 0.826 \(\pm\) 0.003 & 0.794 \(\pm\) 0.004 & 0.810 \(\pm\) 0.003 \\ \hline \end{tabular}
\end{table} 
Table 3: Experiment results of text classification models. To enhance the statistical robustness of the results, we conducted three additional independent experimental runs of MSA K-BERT. In each run, we split the data into training, validation, and test sets with different ratios while keeping the model architecture consistent. The supplementary experimental results are shown in Table 4. Based on these results, we can conclude that the proposed MSA K-BERT demonstrates significant and robust performance improvements on the IMCS-21 dataset. We randomly select 500 samples from the IMCS-21 test set and conduct inference tests for K-BERT and MSA K-BERT under the same hardware environment (NVIDIA RTX 3060 GPU). We measure the average inference time per sample and throughput, with the results shown in Table 5. Although MSA K-BERT may introduce additional attention computations, its actual runtime efficiency still shows certain advantages. In addition, its parameters (310 M) are fewer than those of K-BERT (340 M), indicating that it has the advantages of being higher efficiency and resource-saving. Our model is an improvement based on K-BERT, which is a model that enhances text representation by using KG. This means that our model still has greater potential for improvement. If we can further optimize the selection and use of KG in the model, it may achieve better results. In summary, our model (MSA K-BERT) is an excellent text classification model that outperforms other models in all evaluation metrics, and it also has potential for further