# Compressing BERT for Binary Text Classification via Adaptive Truncation before Fine-Tuning
## 3 Full BERT Does Not Always Perform the Best
### Truncating BERT for Binary Text Classification  
It is widely accepted that BERT is over-parameterized [20; 21], which motivates us to conduct a series of experiments to empirically study how this would influence BTC tasks that are the focus of this study. As \(\text{BERT}_{\text{base}}\) and \(\text{BERT}_{\text{large}}\) have identical structures but different sizes, we focus only on the former and refer to it as BERT from now on for notation simplicity, unless otherwise specified. Further study on \(\text{BERT}_{\text{large}}\) should be pursued in future work. As illustrated in Figure 3, a model for BTC can be built via truncating BERT in two steps: The first step is to drop the top-\(k\) Transformer layers from the pre-trained BERT and preserve the bottom part as an _encoder_. The second step is to add a _task head_ to the top of the resulting encoder, composed of a fully connected (FC) layer and a sigmoid layer to construct a binary classifier. In this way, when \(k\) ranges from 11 to 0, we obtain 12 different classifiers, which are denoted by \(M_{1}\),..., \(M_{12}\), respectively. To train such a classifier \(M_{m}(m=1,\dots,12)\), the parameters in different parts are initialized differently: those in the encoder, i.e., the bottom Transformer layers retained, are initialized using their weights learned in pre-training, whereas those in the task head are randomly initialized. After that, \(M_{m}\) is fine-tuned using labeled datasets of downstream BTC tasks. To investigate the performance of these truncated models, we fine-tune and test them on the datasets of three different BTC tasks:  
* **ChnSentiCorpHou**[48] is a Chinese sentiment classification corpus consisting of hotel comments collected by [PERSON] and [PERSON] [48]. It consists of 3000 positive and 3000 negative reviews of different hotels. Given a comment, the task is to predict the corresponding sentiment polarity. * **CoLA**[49] consists of 10,657 English sentences from published linguistics literature. Given a sentence, the task is to judge the grammatical acceptability of it. Note that, as the test set is unlabeled, we only utilize publicly available training and dev sets in our experiments. * **RTE**[22] refers to the Recognizing Textual Entailment dataset, consisting of 2.7 k English sentence pairs. Each pair is labeled with a binary label indicating whether one sentence entails the other or not. For fairness, different truncated models adopt the same splitting of training (or rather, fine-tuning), validation and test set. The test accuracy each model achieved on these three datasets is given in Table 1, which is evaluated by  
\[accuracy=\frac{\#TP+\#TN}{\#TP+\#TN+\#FP+\#FN} \tag{1}\]  
where \(\#TP\), \(\#TN\), \(\#FP\) and \(\#FN\) denote the number of true positives, true negatives, false positives and false negatives, respectively. Figure 3: The main stages involved in building a model using truncated BERT for a BTC task. The left is the full BERT model, the middle is the truncated BERT after dropping the top-\(k\) Transformer layers of full BERT, and the right is a binary classification model consisting of an encoder (the truncated BERT) and a task-specific head (a fully connected layer followed by a sigmoid layer). Note that, as each Transformer has more than 9M parameters, fine-tuning and inference times would dramatically increase as the number of Transformer layers grows. However, based on the results in Table 1, it is evident that _layers seem to come with diminishing returns in terms of classification accuracy_. As more layers are added, the accuracy fluctuates and then plateaus. More concretely, for CoLA, truncated models with 9 or 10 Transformer layers outperform the full BERT model, i.e., \(M_{12}\), and models with 8 and 11 Transformer layers exhibit competitive performance; for RTE, a model containing 8 Transformers has the best test accuracy; and for ChnSentiCorpHou, no truncated model outperforms the full BERT model, but those with 2 to 11 Transformers are competitive.