# Compressing BERT for Binary Text Classification via Adaptive Truncation before Fine-Tuning
## 5 Experiments
### Discussion  
Overall, the above results show that our method can achieve an averaged compression ratio of about 40% relative to the full BERT while keeping the classification accuracy on a very high level. Specifically, our method achieved the highest accuracy among all models, including the full BERT, on five out of eight datasets, and beat other compressed models on six out of eight datasets while being the runner-up and third winner on the remaining two datasets. Moreover, our method is simple to implement, requires very little additional cost for compression, only needs to fine-tune a truncated BERT with fewer Transformers, and thus enjoys much shorter training times compared with the full BERT. This is impossible for other conventional compression methods, including distillation and pruning, because they need to transfer knowledge from teacher to student or learn the pruning weights/masks in addition to fine-tuning. This generally leads to significant increases in the time and space consumption of the training process. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Model** & **ChnSentiCorpHou** & **WaiMai** & **SST-2** & **CoLA** & **Nikon-JD** & **AFQMC** & **MRPC** & **RTE** \\ \hline Full BERT & 102 & 102 & 109 & 109 & 102 & 102 & 109 & 109 \\ \hline Distil-Pretrain & - & - & - & - & - & - & - \\ Distil-task-biLSTM & 102 + 9 & 102 + 9 & 109 + 9 & 109 + 9 & 102 + 9 & 102 + 9 & 109 + 9 & 109 + 9 \\ Distil-task-kimCNN & 102 + 7 & 102 + 7 & 109 + 7 & 109 + 7 & 102 + 7 & 109 + 7 & 109 + 7 \\ Pruning-AGF & 102 & 102 & 109 & 109 & 102 & 102 & 109 & 109 \\ Truncated-CSM & **53** & **53** & **95** & **88** & **95** & **53** & **60** & **81** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Number of parameters in the model needs to be stored and updated during training/fine-tuning for the 8 BTC tasks in this study. All values have been scaled by 1M. Bold denotes the smallest values. The quantity of Distil-Pretrain is not available, and thus also not shown in the table. The quantity of a distillation based model counts the parameters in both the teacher and student. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Model** & **ChnSentiCorpHou** & **WaiMai** & **SST-2** & **CoLA** & **Nikon-JD** & **AFQMC** & **MRPC** & **RTE** \\ \hline Full BERT & 347 & 439 & 3918 & 334 & 104 & 1617 & 159 & 177 \\ \hline Distil-Pretrain & - & - & - & - & - & - & - \\ Distil-task-biLSTM & 347 + 841 & 439 + 613 & 3918 + 5493 & 334 + 761 & 104 + 248 & 1617 + 3933 & 159 + 371 & 177 + 284 \\ Distil-task-kimCNN & 347 + 738 & 439 + 570 & 3918 + 5052 & 1312 & 104 + 455 & 1617 + 3836 & 1113 & 177 + 261 \\ Pruning-AGF & 10,516 & 5175 & 21,389 & 2326 & 1970 & 16,643 & 2467 & 2004 \\ Truncated-CSM & **158** & **202** & **3286** & **268** & **96** & **771** & **83** & **122** \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Training time (s) for the 8 BTC tasks in this study. Bold denotes the shortest times. For distillation-based method, it includes fine-tuning time of teacher model and distillation time of student model. For pruning methods, it includes fine-tuning time and weight pruning time, whereas for truncation-based method, it includes calculation time of separability measure and fine-tuning time. Note that the total training time of Distil-Pretrain is not available, and hence not given here. Currently, we only concentrate on BTC tasks. Although these tasks can support a long list of practical applications, for greater real-world applicability, they need to be formulated into more complicated tasks, such as multi-class classification and sequence labeling. As the separability indices also work for multi-class classification, we believe the work presented in this study could provide a basis for future studies on compression methods for more complex (downstream) tasks.