# Compressing BERT for Binary Text Classification via Adaptive Truncation before Fine-Tuning
###### Abstract  
Large-scale pre-trained language models such as BERT have brought much better performance to text classification. However, their large sizes can lead to sometimes prohibitively slow fine-tuning and inference. To alleviate this, various compression methods have been proposed; however, most of these methods solely consider reducing inference time, often ignoring significant increases in training time, and thus are even more resource consuming. In this article, we focus on lottery ticket extraction for the BERT architecture. Inspired by observations that representations at lower layers are often more useful for text classification, we propose that we can identify the winning ticket of BERT for binary text classification through adaptive truncation, i.e., a process that drops the top-\(k\) layers of the pre-trained model based on simple, fast computations. In this way, the cost for compressing and fine-tuning, as well as inference, can be vastly reduced. We present experiments on eight mainstream binary text classification datasets covering different input styles (i.e., single-text and text-pair), as well as different typical tasks (e.g., sentiment analysis, acceptability judgement, textual entailment, semantic similarity analysis and natural language inference). Compared with some strong baselines, our method saved 78.1% time and 31.7% memory on average, and up to 86.7 and 48% in extreme cases, respectively. We also saw good performance, often outperforming the original language model. model compression; binary text classification; lottery ticket hypothesis; layer truncation; fine-tuning +
Footnote †: journal: Applied Sciences  
From the perspective of downstream tasks, current model compression methods could be classified as task-agnostic and task-specific compression. The former aims at compressing pre-trained language models for any downstream NLP task, whereas the latter shrinks models that are fine-tuned for a particular task. As discussed in [8; 14; 15], task-specific compression has two important advantages over task-agnostic compression: First, it is often faster and more sample-efficient. In contrast, task-agnostic compression requires (re-)training on very large (unlabeled) datasets, and hence is resource-hungry. Second, models obtained by task-specific compression can adopt specialized attention patterns that are useful for downstream tasks, but not necessarily useful for general language modeling. Contemporary task-specific compression methods fall into four categories: quantization, pruning, distillation and matrix decomposition [8; 16; 17; 18]. Among them, distillation and pruning are perhaps the most used. The general procedure of these two kinds of methods is illustrated in Figure 1 (upper box). (Note that pruning methods commonly require fine-tuning again after model compression, which has been omitted in Figure 1 due to this process being very simple and rarely mentioned in the literature.) As shown, knowledge distillation implements compression via transferring knowledge from a large teacher to a much smaller student, whereas the pruning methods finish compression based on one model. There exist two common weaknesses of these methods: First, they reduce the calculations needed in the inference stage at the cost of a much longer training time (fine-tuning and compressing). In fact, the resources consumed by fine-tuning are often prohibitive of the application of these methods [6; 7]. Second, few of them consider the complexity of downstream tasks, as well as how the language models and target tasks correlate, when determining the architecture of the student model or the units in the language model to be pruned. To address these weaknesses, we present a new task-specific model compression method and apply it to the BERT language model [4], which is also illustrated in Figure 1 (lower box) for ease of comparison.