## 5 Experiment
### Intrinsic Evaluation 2: Evaluate SEMTS-BERT on Text Classification  
Text classification is a popular task used to evaluate the performance of models in NLP [5]. In this section, we evaluate the performances of different sentence representation models on three Chinese text classification datasets.  
* [noitemsep,nolistsep]
* Thucnews: A high-quality 14-category text classification dataset containing 0.74 M news articles collected from Sina News: [[https://news.sina.com.cn/](https://news.sina.com.cn/)]([https://news.sina.com.cn/](https://news.sina.com.cn/)), accessed on 1 May 2021. The dataset is provided by the NLP Laboratory of Tsinghua University: [[http://thuctc.thunlp.org/](http://thuctc.thunlp.org/)]([http://thuctc.thunlp.org/](http://thuctc.thunlp.org/)), accessed on 1 May 2021.
* Fudan Dataset (Fudan): Fudan dataset: [[http://www.nlpir.org/?action-viewnews-itemid-103](http://www.nlpir.org/?action-viewnews-itemid-103)]([http://www.nlpir.org/?action-viewnews-itemid-103](http://www.nlpir.org/?action-viewnews-itemid-103)), accessed on 1 May 2021, is a 20-category text classification dataset that contains 9833 test documents and 9804 training documents. It is an uneven dataset since the number of documents of each category varies greatly.
* TouTiao Dataset (TouTiao): TouTiao dataset is a 15-category short text classification dataset. Each example contains only a news headline and a subheading. It is noisy and collected from TouTiao news website: www.toutiao.com, accessed on 1 May 2021.  
\begin{table}
\begin{tabular}{r c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{5}{c}{**Probing Tasks**} & \multirow{2}{*}{**Avg.**} \\ \cline{2-2} \cline{4-6}  & **Sentence\_Len** & **Bshift** & **Voice** & **SubjNum** & \\ \hline \(BERT_{cls}\) & 0.431(0.129) & **0.871(0.188)** & 0.927(0.017) & **0.983(0.007)** & 0.803(0.085) \\ \(BERT_{max}\) & 0.561(0.136) & 0.779(0.110) & 0.972(0.021) & 0.829(0.125) & 0.785(0.098) \\ \(BERT_{mem}\) & 0.516(0.159) & 0.687(0.200) & 0.981(0.008) & 0.825(0.253) & 0.752(0.155) \\ \(fasttext_{cls}\) & 0.825(0.005) & 0.505(0.002) & 0.951(0.002) & 0.957(0.002) & 0.810(0.003) \\ \(fasttext_{max}\) & 0.781(0.006) & 0.497(0.002) & 0.926(0.001) & 0.963(0.002) & 0.792(0.003) \\ \(fasttext_{mem}\) & 0.804(0.006) & 0.513(0.002) & 0.950(0.001) & 0.963(0.003) & 0.808(0.003) \\ \(sent2 vec\) & 0.861(0.005) & 0.509(0.001) & **0.983(0.003)** & 0.965(0.003) & 0.830(0.003) \\ \(LASER\) & 0.938(0.005) & 0.563(0.006) & 0.950(0.002) & 0.967(0.002) & 0.855(0.004) \\ \(SBERT\)-_WK_ & 0.936(0.006) & 0.731(0.004) & 0.938(0.005) & 0.962(0.001) & 0.892(0.004) \\ \(SEMTS\)-_BERT_ & **0.959(0.001)** & 0.740(0.005) & 0.956(0.003) & 0.972(0.002) & **0.907(0.003)** \\ \hline \hline \end{tabular}
\end{table}
Table 2: The experimental results of different models in probing tasks.  
We first split paragraphs into sentences and the calculate the representations of sentences. We later use a BiLSTM encoder to encode the sentence representation sequence. Finally, the outputs of the encoder (document representations) are inputted into the softmax layer to perform text classification. The architecture of the text classifier is shown in Figure 8. We use the Adam optimizer [35] to train each model 50 epochs with early stop strategy on every dataset.