# Compressing BERT for Binary Text Classification via Adaptive Truncation before Fine-Tuning
###### Abstract
As shown, the proposed method relies on adaptively  
Figure 1: The general procedures of conventional task-specific compression (**upper** box) and the adaptive truncation compression proposed in this article (**lower** box). Conventional methods commonly conduct compression after fine-tuning, whereas ours conducts compression prior to fine-tuning. Moreover, knowledge distillation requires an additional training process for the teacher to teach the student to make predictions. Pruning methods are needed evaluate the importance of each model unit or learned weight to determine which should be masked and then perform another round of fine-tuning; our method demands no additional training. truncating BERT for a given BTC task before fine-tuning. Specifically, by dropping the top-\(k\) Transformer layers in BERT, we observe that the full BERT is not always the best choice when fine-tuning on downstream BTC tasks. The classification accuracy achieved by some truncated ones can get very close to or even higher than that of the full BERT. This indicates that compressing BERT via appropriate truncation could cause no significant loss and even bring notable gain to the prediction accuracy of downstream BTC tasks. To properly and efficiently determine the truncation position, we conduct some theoretic analysis of the fine-tuning process, together with a series of tentative experiments, which motivate us to choose a class separability index as a proxy. The chosen measure can be rapidly calculated before fine-tuning and is found to be highly correlated with the final classification accuracy. As a result, only truncated BERT, plus a task-specific head, needs to be fine-tuned, and hence resources consumed in both fine-tuning and inference stages could be saved with negligible additional cost. For validation, we conduct experiments on eight commonly used BTC tasks. The results show that our method, albeit simple, is very effective, outperforming both knowledge distillation and weight pruning methods. In summary, the main contributions of this article include:  
* Proposition of a simple, but effective model compression method that adaptively truncates BERT by layer-dropping prior to fine-tuning, substantially reducing the cost of fine-tuning and inference. We observe that the full BERT model is not always the best point of departure for fine-tuning models for particular downstream tasks. Sometimes shrinking BERT by truncating layers leads to better performance, as well as faster fine-tuning and inference. * A criterion to estimate truncating position, according to pre-trained representations of instances. As a result, our method finds winning tickets [19; 20; 21] more before fine-tuning. In addition, the provided criterion makes the balance between the loss in classification accuracy and reduction of the model size controllable, and thus is adaptive for downstream tasks and supports compression on-demand. * Experiments that show that, compared with some strong baselines, the proposed method reduces fine-tuning time by 78% and memory requirements by 31.7%, on average, with up to 86.7 and 48% reductions in some extreme cases, respectively. On average, 39.6% of storage can be saved compared with original BERT-based models in both fine-tuning and inference. In addition, the accuracy achieved surpasses all our baselines on six out of eight tasks, and is higher than the full BERT model on five tasks. The remainder of this article is structured as follows: Section 2 briefly reviews related work; Section 3 discusses how to truncate BERT and how the resulting models perform on BTC tasks; Section 4 presents our compression method of truncating BERT before fine-tuning; Section 5 describes the experimental settings, the results of our approach and competitive methods. Finally, we draw a conclusion and evaluate future work in Section 6.