## 5 Experiment
### Intrinsic Evaluation 3: Evaluate SEMTS-BERT on Natural Language Inference (NLI)
#### 5.7.4 Experimental Results  
The comparisons between our model and other models on BQ and LCQMC are shown in Tables 6 and 7, Appendices A and B. \"c\" and \"w\" in the \(Emb\) column represent the character-based and the word-based model. \"Acc.\" represents the classification accuracy. \"+st.\" and \"+dy.\" denote that word embeddings are fixed and to be tuned during the training. \"+LASER\", \"+SB-WK\", and \"+SEMIS\" mean that the embeddings used to replace the original embeddings of low-frequency words are calculated by LASER, SBERT-WK, and SEMIS-BERT, respectively. On both BQ and LCQMC, we all use the word-based BiMPM. On the BQ dataset, we obtain the similar performance to the benchmark obtained by a character-based model [37]. Generally, the performance of character-based model is better than the performance of word-based model on small Chinese datasets due to sparsity [16]. On the LCQMC dataset, we achieve better performance than the benchmark. From Tables 6 and 7, Appendices A and B, we can draw a conclusion that we achieve better performance when we replace the low-frequency words' embeddings on the larger LCQMC dataset. The performance of the word-based model exceeds that of the character-based model after performing the replacement. This shows that we can obtain higher-quality word embeddings through the proposed method when the dataset is large. On the smaller BQ dataset, the replacement also promotes the word-based model. Sometimes, the performance of word-based models exceeds that of the character-based model after performing the replacement. In summary, the performance improvement indicates that our method can provide higher-quality low-frequency words' embeddings. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Emb** & **P** & **R** & **F** & **Acc.** \\ \hline _TF-IDF_ & c & 64.68 & 60.94 & 62.75 & 63.83 \\ \hline _Text-CNN_[44] & c & 67.77 & 70.64 & 69.17 & 68.52 \\ _Text-CNN_[44] & w & 69.61 & 67.00 & 68.28 & 67.56 \\ _Text-CNN_[44]+st.+LASER & w & 68.96 & 68.38 & 68.67 & 68.30 \\ _Text-CNN_[44]+dy.+LASER & w & 68.63 & 67.09 & 67.85 & 68.87 \\ _Text-CNN_[44]+st.+SB-BK & w & 67.82 & 69.42 & 68.61 & 68.22 \\ _Text-CNN_[44]+dy.+SB-BK & w & 68.38 & 67.70 & 68.04 & 67.58 \\ _Text-CNN_[44]+st.+SEMIS & w & 67.91 & 70.15 & 69.01 & 68.49 \\ _Text-CNN_[44]+dy.+SEMIS & w & 68.65 & 69.01 & 68.83 & 67.37 \\ \hline _BiLSTM_[48] & c & 75.04 & 70.46 & 72.68 & 73.51 \\ _BiLSTM_[48] & w & 74.79 & 68.52 & 71.52 & 71.06 \\ _BiLSTM_[48]+st.+LASER & w & 72.68 & 74.28 & 73.47 & 73.55 \\ _BiLSTM_[48]+dy.+LASER & w & 73.01 & 73.39 & 73.20 & 72.73 \\ _BiLSTM_[48]+st.+SB-BK & w & 72.85 & 72.73 & 72.79 & 73.28 \\ _BiLSTM_[48]+dy.+SB-BK & w & 72.63 & 73.09 & 72.86 & 72.66 \\ _BiLSTM_[48]+st.+SEMIS & w & 73.94 & 72.83 & 73.38 & 73.45 \\ _BiLSTM_[48]+dy.+SEMIS & w & 72.86 & 73.10 & 72.98 & 72.89 \\ \hline _DIN_[45] & c & 81.58 & 81.14 & 81.36 & 81.41 \\ _DIN_[45] & w & 81.71 & 79.23 & 80.45 & 80.78 \\ _DIN_[45]+st.+LASER & w & 81.50 & 81.16 & 81.33 & 81.27 \\ _DIN_[45]+dy.+LASER & w & 80.97 & 81.33 & 81.15 & 80.85 \\ _DIN_[45]+st.+SB-BK & w & 81.02 & 81.50 & 81.26 & 81.29 \\ _DIN_[45]+dy.+SB-BK & w & 81.91 & 79.86 & 80.87 & 80.75 \\ _DIN_[45]+st.+SEMIS & w & 81.07 & 81.63 & 81.35 & 81.39 \\ _DIN_[45]+dy.+SEMIS & w & 81.56 & 80.11 & 80.83 & 80.91 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: The comparison between our method and other models on BQ. Figures 11 and 12 show the experimental results on BQ and LCQMC at different low-frequency word embedding replacement rates. The bars \"SEMTS-BERT *\" represent the accuracy of BiMPM at different low-frequency word embedding replacement rates when SEMTS-BERT is used to estimate the embeddings. The bars \"LASER *\" and \"SBERT-WK ** also have similar meanings. \"Static\" and \"dynamic\" indicate that word embeddings are fixed and to be tuned during the training. When the low-frequency word embedding