# Compressing BERT for Binary Text Classification via Adaptive Truncation before Fine-Tuning
## 4 Truncating BERT before Fine-Tuning
### Feasibility Analysis  
Recall that, in Section 3, we constructed 12 binary classifiers, denoted by \(M_{1}\),..., \(M_{12}\), each of which contains a truncated BERT with \(m\) (\(=1,\ldots,12\)) Transformer layers, followed by an FC layer and a sigmoid layer. More formally, for a given sample (e.g., a text or text-pair) vectorized as \(\mathbf{x}_{n}\), the prediction function of \(M_{m}\) is  
\[M_{m}(\mathbf{x}_{n})=\sigma(g_{m}(h_{m}(\mathbf{x}_{n}))), \tag{2}\]  
\[h_{m}(\mathbf{x}_{n})=f_{m}(f_{m-1}(\ldots f_{1}(\mathbf{x}_{n})\ldots)), \tag{3}\]  
where \(\sigma\) is the sigmoid function, \(g_{m}(\cdot)\) denotes the transform function of a fully-connected layer, \(h_{m}(\cdot)\) is the function of the encoder, and \(f_{j}(\cdot)\) is the function of the \(j\)-th (\(j=1,\ldots,m\)) Transformer layer. For binary classification tasks, \(\sigma(g_{m}(\cdot))\) in Equation (2) is a generalized linear transform targeted to find a hyperplane best separating examples without transforming the \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Classifiers** & **ChnSentiCorpHou** & **CoLA** & **RTE** \\ \hline \(M_{1}\) & 88.08 & 69.07 & 52.00 \\ \(M_{2}\) & 91.08 & 66.22 & 52.29 \\ \(M_{3}\) & 91.52 & 69.64 & 56.38 \\ \(M_{4}\) & 92.07 & 72.87 & 63.06 \\ \(M_{5}\) & 92.65 & 75.52 & 64.73 \\ \(M_{6}\) & 92.45 & 77.80 & 63.57 \\ \(M_{7}\) & 93.05 & 77.99 & 64.29 \\ \(M_{8}\) & 92.07 & 80.83 & **66.03** \\ \(M_{9}\) & 93.28 & 84.25 & 63.68 \\ \(M_{10}\) & 92.02 & **85.96** & 63.71 \\ \(M_{11}\) & 93.02 & 83.49 & 64.94 \\ \(M_{12}\) & **93.30** & 83.87 & 65.81 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Test accuracy (%) of \(M_{1}\),..., \(M_{12}\) on three BTC datasets. Each row corresponds to a classifier with a truncated BERT with a different number of Transformers as its encoder. Column 2–4 are the classification accuracies these classifiers achieve on the three datasets. Bold denotes the highest accuracy. input representations, namely the embedding vectors of the input examples. Such embeddings are produced by the encoder \(h_{m}(\cdot)\), which plays a more critical role as it determines the space that the separating hyperplane is located in and the feature vectors of the input examples, and hence the degree of how the examples are (linearly) separable. More precisely, the prediction accuracy of \(M_{m}(\cdot)\), denoted by \(acc_{m}\), on a dataset \(\mathbb{S}\) containing \(N\) examples vectorized as \(\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\), relies mainly on the linear separability of the embedding set \(\{h_{m}(\mathbf{x}_{1}),\ldots,h_{m}(\mathbf{x}_{N})\}\) generated by the encoder \(h_{m}(\cdot)\). The higher the separability of the embeddings set, the higher the test accuracy \(acc_{m}\) attained by the model \(M_{m}(\cdot)\). In addition, for BTC tasks, we observed relatively small changes were brought by the fine-tuning process to the parameters of the encoder \(h_{m}(\cdot)\) in our experiments, particularly for the deeper layers. We owe this to three main factors: (i) the labeled dataset used for fine-tuning is of a small size, commonly ranging from hundreds to tens of thousands, (ii) the vanishing effect caused by computing the gradients of model parameters by the chain rule and (iii) the often very slow learning rate, e.g., \(5\times 10^{-5}\), applied to the pre-trained encoder. Similar phenomena have been previously observed by [PERSON] et al. [50] on the MNLI dataset [31]. Even the very recent studies conducted by [PERSON] and [PERSON] [51; 52] found that fine-tuning largely preserved the spatial structure of the data points, i.e., the vectorized examples. Inspired by the above analyses and observations, we conjecture that the ordering of the separability of the embedding sets generated by different pre-trained encoders remains consistent during fine-tuning.