## 5 Experiment
### Intrinsic Evaluation 3: Evaluate SEMTS-BERT on Natural Language Inference (NLI)
#### 5.7.4 Experimental Results
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Emb** & **P** & **R** & **F** & **Acc.** \\ \hline _BiMPM_[46] & c & 82.28 & 81.18 & 81.73 & **81.85** \\ _BiMPM_[46] & w & 81.35 & 81.11 & 81.22 & 81.28 \\ _BiMPM_[46]+st.+LASER & w & 81.10 & 82.31 & 81.70 & 81.15 \\ _BiMPM_[46]+dy.+LASER & w & 80.85 & 82.20 & 81.52 & 80.86 \\ _BiMPM_[46]+st.+SB-BK & w & 80.93 & 82.44 & 81.68 & 81.18 \\ _BiMPM_[46]+dy.+SB-BK & w & 81.09 & 81.73 & 81.41 & 80.92 \\ _BiMPM_[46]+st.+SEMTS & w & 82.16 & 81.30 & 81.73 & 81.77 \\ _BiMPM_[46]+dy.+SEMTS & w & 80.59 & 81.45 & 81.02 & 81.13 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: _Cont._ \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Emb** & **P** & **R** & **F** & **Acc.** \\ \hline _CBOW_[49] & c & 66.5 & 82.8 & 73.8 & 70.6 \\ _CBOW_[49] & w & 67.9 & 89.9 & 77.4 & 73.7 \\ _CBOW_[49]+st.+LASER & w & 67.75 & 70.64 & 77.65 & 75.05 \\ _CBOW_[49]+dy.+LASER & w & 68.21 & 70.64 & 77.51 & 74.93 \\ _CBOW_[49]+st.+SB-BK & w & 68.28 & 70.64 & 78.07 & 75.11 \\ _CBOW_[49]+dy.+SB-BK & w & 68.53 & 70.64 & 77.64 & 74.98 \\ _CBOW_[49]+st.+SEMTS & w & 68.24 & 70.64 & 78.13 & 75.37 \\ _CBOW_[49]+dy.+SEMTS & w & 67.65 & 70.64 & 77.92 & 75.16 \\ \hline _Text-CNN_[44] & c & 67.1 & 85.6 & 75.2 & 71.8 \\ _Text-CNN_[44] & w & 68.4 & 84.6 & 75.7 & 72.8 \\ _Text-CNN_[44]+st.+LASER & w & 69.45 & 84.04 & 76.05 & 73.85 \\ _Text-CNN_[44]+dy.+LASER & w & 67.76 & 86.31 & 75.92 & 73.56 \\ _Text-CNN_[44]+st.+SB-BK & w & 70.84 & 82.62 & 76.28 & 73.90 \\ _Text-CNN_[44]+dy.+SB-BK & w & 68.37 & 85.93 & 76.15 & 73.51 \\ _Text-CNN_[44]+st.+SEMTS & w & 69.93 & 85.68 & 77.01 & 73.98 \\ _Text-CNN_[44]+dy.+SEMTS & w & 69.61 & 85.87 & 76.89 & 73.54 \\ \hline _BiLSTM_[50] & c & 67.4 & 91.0 & 77.5 & 73.50 \\ _BiLSTM_[50] & w & 70.6 & 89.3 & 78.9 & 76.10 \\ _BiLSTM_[48]+st.+LASER & w & 70.97 & 89.49 & 79.16 & 77.14 \\ _BiLSTM_[48]+dy.+LASER & w & 71.16 & 88.83 & 79.02 & 77.02 \\ _BiLSTM_[48]+st.+SB-BK & w & 70.93 & 89.37 & 79.09 & 76.95 \\ _BiLSTM_[48]+dy.+SB-BK & w & 71.25 & 88.26 & 78.85 & 76.87 \\ _BiLSTM_[48]+st.+SEMTS & w & 71.07 & 89.84 & 79.36 & 77.31 \\ _BiLSTM_[48]+dy.+SEMTS & w & 71.51 & 88.59 & 79.14 & 77.18 \\ \hline _BiMPM_[46] & c & 77.60 & 93.90 & 85.00 & 83.40 \\ _BiMPM_[46] & w & 77.70 & 93.50 & 84.90 & 83.30 \\ _BiMPM_[46]+st.+LASER & w & 78.07 & 96.17 & 86.18 & 84.89 \\ _BiMPM_[46]+dy.+LASER & w & 77.98 & 96.76 & 86.36 & 85.11 \\ _BiMPM_[46]+st.+SB-BK & w & 78.02 & 96.62 & 86.33 & 85.05 \\ _BiMPM_[46]+dy.+SB-BK & w & 78.87 & 94.18 & 85.85 & 84.88 \\ _BiMPM_[46]+st.+SEMTS & w & 79.20 & 95.53 & 86.60 & **85.17** \\ _BiMPM_[46]+dy.+SEMTS & w & 78.95 & 94.12 & 85.87 & 84.85 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: The comparison between our method and other models on LCQMC. replacement rate is 0, we did not use the word embeddings calculated by the sentence representation model to replace the original word embeddings estimated by Word2 Vec. It can be seen from Figure 11 that when using LASER and SBERT-WK to estimate OOVs and low-frequency words' embeddings, the replacement cannot improve the accuracy of BiMPM on BQ, no matter if the embeddings of words are static or dynamic. When using SEMTS-BERT to estimate OOVs and low-frequency words' embeddings and the embeddings of words are fixed during the evaluation, the replacement of low-frequency words' embeddings can improve the accuracy of the word-based BiMPM at the replacement rate of 1.95% (from 81.28% to 81.77%). When word embeddings are dynamic, SEMTS-BERT can not improve the performance of the word-based BiMPM on BQ either. Figure 11: The influence of different low-frequency word embedding replacement rates on the performance of BiMPM evaluated on BQ. Figure 12: The influence of different low-frequency word embedding replacement rates on the performance of BiMPM evaluated on LCQMC. The situation in Figure 12 is slightly different. On LCQMC, whether the word embeddings are dynamic or static, replacing low-frequency words' embeddings can improve the performance of the word-based BiMPM.