# Compressing BERT for Binary Text Classification via Adaptive Truncation before Fine-Tuning
## 4 Truncating BERT before Fine-Tuning
### Using Class Separability as a Proxy  
We conducted additional experiments to validate the conjecture described above. Before that, three widely used separability indices, viz., class scatter matrices (CSM) [53], [PERSON]'s separability index (SI) [54] and hypothesis margin (HM) [55], are briefly reviewed in the following section. To simplify notation, the embedding vector generated by \(h_{m}(\cdot)\) from an input example \(\mathbf{x}\) is denoted by \(\mathbf{z}\) from now on, viz., \(\mathbf{z}=h_{m}(\mathbf{x}),\ m=1,\ldots,12\), in which the model subscript is omitted.  
**CSM** is the fraction of the trace of between-cluster scatter matrix (\(S_{B}\)) and within-cluster scatter matrix (\(S_{W}\)). It could be formulated as follows for two-class datasets:  
\[CSM=tr(S_{B})/tr(S_{W}), \tag{4}\]  
\[S_{B}=\sum_{c=1}^{2}N_{c}(\overline{\mathbf{z}_{c}}-\overline{\mathbf{z}})( \overline{\mathbf{z}_{c}}-\overline{\mathbf{z}})^{\text{T}},\]  
**HM** evaluates the sum of the distance between the hypothesis and the closest hypothesis that assigns the alternative label to the given instance.  
\[HM=\frac{1}{2}\sum\
olimits_{i=1}^{N}(\parallel\mathbf{z}_{i}-\text{nearmiss}( \mathbf{z}_{i})\parallel-\parallel\mathbf{z}_{i}-\text{nearhit}(\mathbf{z}_{i}) \parallel), \tag{8}\]  
where \(N\) denotes the number of instances in the dataset, \(\text{nearmiss}(\mathbf{z}_{i})\) and \(\text{nearhit}(\mathbf{z}_{i})\) are the nearest instances to \(\mathbf{z}_{i}\) with the same and different class labels, respectively, and \(\parallel\cdot\parallel\) stands for the Euclidean norm.  
To evaluate the appropriateness of these three separability indices for use as proxies for truncating BERT before fine-tuning, we computed them on the embedding sets generated by the pre-trained (but not fine-tuned) encoders, \(h_{1}(\cdot),\ldots,h_{12}(\cdot)\), from the three BTC datasets described in Section 3.2. Figure 4 shows the results. For the convenience of comparison, we draw the results of one separability index together with the classification accuracy of the corresponding classifiers in each subplot, i.e., \(M_{1}\), \(\ldots,M_{12}\), on the same dataset. Intuitively, the trends of CSM, compared with SI and HM, are more similar to that of test accuracy across encoders and datasets. As shown in the first column in Figure 4, CSM has similar test accuracy to that achieved by the corresponding classifiers after fine-tuning. This somewhat validates our conjecture presented at the end of Section 4.1. We further compute the Pearson's correlation coefficient between classification accuracy and each separability index. The results are given in Table 2, which quantitatively confirm that CSM most closely correlates with test accuracy, and hence is the best choice among the three indices to be used as a proxy.  
Figure 4: The test accuracy (black) vs. CSM (red), SI (blue) and HM (green) curves on 3 BTC datasets. We juxtapose the separability index and classification accuracy for all 12 Transformer layers. The horizontal axis is the layer index of BERT, the left vertical axis is the classification accuracy and the right vertical axis is the separability index. Each row corresponds to a different dataset, and each column corresponds to a different separability index. Best viewed in color.