# Compressing BERT for Binary Text Classification via Adaptive Truncation before Fine-Tuning
## 5 Experiments
### Results and Analysis  
#### 5.3.1 The Balance between Accuracy Decline and Model Size Reduction  
We investigate how the classification accuracy and model size vary along with different \(\rho\) (maximally tolerable ratio of relative accuracy decline) to find an appropriate trade-off between them. To check whether there exists some general regularity, we first treat the aforementioned eight BTC tasks as a whole by computing the average accuracy loss and compression ratio on them for different \(\rho\). Figure 5 depicts how the average accuracy loss (measured using the mean accuracy loss ratio relative to the full BERT; see the left vertical  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Corpus** & **Batch Size** & **Learning Rate** & **Max Seq. Length** & **Device** \\ \hline ChnSentiCorpHu & 32 & \(1\times 10^{-5}\) & 512 & RTX3090 \(\times\) 2 \\ WaiMai & 32 & \(1\times 10^{-5}\) & 256 & RTX3090 \(\times\) 2 \\ SST-2 & 32 & \(2\times 10^{-5}\) & 128 & V100 \(\times\) 1 \\ CoLA & 32 & \(2\times 10^{-5}\) & 128 & RTX3090 \(\times\) 2 \\ Nikon-JD & 32 & \(1\times 10^{-5}\) & 256 & RTX3090 \(\times\) 2 \\ AFQMC & 32 & \(5\times 10^{-6}\) & 256 & RTX3090 \(\times\) 2 \\ MRPC & 32 & \(1\times 10^{-5}\) & 128 & RTX3090 \(\times\) 2 \\ RTE & 32 & \(1\times 10^{-5}\) & 256 & V100 \(\times\) 1 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Parameter settings in our implementations.  
axis) and average compression ratio achieved (measured using the average number of Transformers dropped during compression; see the right vertical axis) vary when \(\rho\) changes from 0 to 1 with a stride of 0.05. It can be observed that both accuracy loss and compression ratio increase when \(\rho\) grows larger. This is consistent with our expectations. Notably, when \(\rho>0.40\), the mean accuracy loss ratio (see the red curve in Figure 5) starts accelerating going up, whereas the increase in compression ratio (see the blue curve in Figure 5) slows down. Both curves are nearly horizontal in the interval of \(\rho\in[0.35,\ 0.40]\); for a better trade-off, we would limit \(\rho\leq 0.35\). If we prefer near to no accuracy loss, we may limit the mean loss ratio to be less than 0.02 (i.e., within 2%). Correspondingly, \(\rho\) should be no more than 0.25. Based on the figure, it is obvious that \(\rho\) should be limited to \([0.20,\ 0.25]\) if we want to obtain a compression ratio as large as possible at the same time. In addition, as the red curve is steeper than the blue one when \(\rho\in[0.20,\ 0.25]\), _we think \(\rho=0.20\) is an even better choice, which restricts accuracy loss more rigidly while achieving a close compression ratio_.  
Figure 6 further depicts how the test accuracy achieved (the left vertical axis in each subplot) and number of Transformer layers retained (the right vertical axis in each subplot) covary with different \(\rho\) on each dataset. We highlight the test accuracy and size of compressed model corresponding to \(\rho=0.20\) using a vertical dashed line in each subplot. The results show that this \(\rho\) setting helps to find a compressed model with the highest accuracy or accuracy very close to the highest on all datasets but AFQMC. This verifies \(\rho=0.20\) to be a generally good choice. Moreover, the resulting model is of different complexity, as it contains a different number of Transformers for different BTC datasets. In other words, _the number of Transformers to be preserved in the encoder is adaptively determined for the downstream BTC task_.