# Compressing BERT for Binary Text Classification via Adaptive Truncation before Fine-Tuning
## 5 Experiments
### Results and Analysis
#### 5.3.2 Performance of the Compressed Models  
To validate the effectiveness of our proposed method, we comprehensively evaluate the performance of the compressed model it produces when adopting an optimal balance between accuracy loss and compression ratio, namely with a setting of \(\rho=0.2\). The performance is measured in different angles including the test accuracy achieved, the number of parameters remained and the training time spent. The results are shown in Tables 5-7, respectively, in which our compressed model is called Truncated-CSM. For the convenience of comparison, we also provide the results of the baseline models on identical datasets in these tables, using the same devices as those listed in Table 4.  
As shown in Table 5, compared with other compressed models, ours achieves the highest accuracy on six out of eight datasets. For the other two datasets, it wins the second best on AFQMC and the third best on MRPC, and performs very closely to Pruning-AGP in that the accuracy difference between them is no more than one point. It is also noteworthy  
Figure 5: The mean accuracy loss ratio (the left vertical axis) and the number of dropped Transformer layers (the right vertical axis) vary when \(\rho\) changes from 0 to 1 with a stride of 0.05. The horizontal axis is the value of \(\rho\). The vertical dashed line corresponds to the upper bound of \(\rho\) that can bring good trade-offs between accuracy loss and compression ratio. The gray interval corresponds to \(\rho\in[0.20,\ 0.25]\), which can achieve the largest compression ratio without notable accuracy loss. Best viewed in color. that though our model is often much smaller than the full BERT (see the corresponding columns in Table 6), it outperforms the full BERT on five datasets in terms of test accuracy. This shows that BERT is over-parameterized, as previously found, and justifies our method of compressing it via truncation. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline
**Model** & **ChSentiCorpHow** & **WaiMai** & **SST-2** & **CoLA** & **Nikon-JD** & **AFQMC** & **MRPC** & **RTE** \\ \hline Full BERT & **93.30** & 90.12 & 92.26 & 83.87 & 97.01 & **74.12** & **83.59** & 65.81 \\ \hline Distil-Pretrain & 91.17 & 89.82 & 91.71 & 81.40 & 90.72 & 51.88 & 82.09 & 59.56 \\ Distil-task-bILSTM & 87.50 & 89.57 & 87.10 & 68.31 & 89.32 & 64.18 & 64.17 & 50.88 \\ Distil-task-kimCNN & 86.48 & 88.45 & 85.23 & 70.59 & 91.29 & 63.51 & 67.59 & 52.08 \\ Pruning-AGF & 92.65 & 87.11 & 91.32 & 82.73 & 95.23 & 69.95 & 77.33 & 57.50 \\ Truncated-CSM & **92.65** & **90.78** & **93.03** & **84.25** & **97.31** & 68.95 & 76.52 & **66.03** \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Test accuracy (%) on the 8 tasks in this article. Bolded values denote the highest among all models (i.e., including full BERT), and underlined values denote the best among compressed models. Figure 6: The test accuracy (left vertical axis) and number of Transformer layers preserved in the truncated model (right vertical axis) vary when \(\rho\) changes from 0 to 1 with a stride of 0.05. The horizontal axis is the value of \(\rho\). Each subplot corresponds to a different dataset. The vertical dashed line in each subplot corresponds to \(\rho=0.20\). Best viewed in color. We further compare the quantities of parameters in different compressed models, together with that of the full BERT, which need to be stored and updated during training. Note that, for inference, distillation-based methods only load and use the student, which is often much lighter and runs very fast. However, as shown in Table 6, such model reduction and speed-up are gained at the cost of far more resources invested into training. When looking back at training, the results in Table 6 demonstrate that our compressed model is much smaller than all competing models, including the full BERT, which leads to significantly less training time. This is indicated by the results in Table 7, showing that our model consumes the least training time on all datasets.