# A Green AI Methodology Based on Persistent Homology for Compressing BERT
## 4 Empirical Evaluation
### Distribution of \(r_{f}\) and Selection of the More Informative Neurons  
The analysis of the distribution of \(r_{f}\) is crucial for correctly identifying which neurons contribute the most information to a neural network's predictions and which units are dispensable. The premise here is that a neuron with a high associated \(r_{f}\) value implies that its outputs exhibit sufficient variability, making it important when the network computes outputs. Bearing in mind that the mean can be heavily influenced by excessively high \(r_{f}\) values in some situations, the median was chosen as the key metric to consider. Figure 6 shows the median of the \(r_{f}\) values for each layer and component of BERT Base (upper image, Figure 6a) and BERT Large (lower image, Figure 6b). The Figure helps determine the pruning intensity to apply to each component and layer of the model. By analyzing the median of the \(r_{f}\) values of the neurons within that component, PBCE quantifies the importance of each component and, consequently, the extent to which it should be pruned. The experimentation revealed that components with \(r_{f}\) values above 0.2 will receive light pruning, values between 0.1 and 0.2 should receive medium pruning, and values below 0.1 should undergo intensive pruning. Nevertheless, it is crucial to asses the optimal cut-off value for each specific case to ensure effective pruning. Before delving into the details of simplification for each architecture, some considerations that apply to both BERT Base and BERT Large should be taken into account. Firstly, PBCE starts the simplification process from the third layer on. This decision minimally reduces compression capacity, but does favor subsequent fine-tuning to meet the requirements of the GLUE benchmark tasks. In addition, Attention Output and Output components' implementation involves a LayerNorm operation in which the state of the data and the original input are added together. As a consequence, these components cannot be simplified, and will be excluded of the analysis. Additionally, it is observed that the behavior of the Q and K components for each BertLayer are very similar, so the same pruning process will be applied to them. Additionally, it has been verified that the component that contributes the most information, according to our hypothesis, is the Intermediate component. Tables 2 and 3 present the distribution of levels of pruning to each component and layer. For example, for the Q component, layers 3, 4, and 12 are lightly pruned (retaining neurons with \(r_{f}\) values higher than the first quartile), layers from 8 to 11 are pruned with medium intensity (retaining the second quartile of neurons), and layers 5, 6, and 7 are severely pruned (retaining the third quartile of neurons). Next, the main ideas derived from Figure 6 are explained. In the case of BERT Base, there is an increasing trend in the contribution of neurons, with outputs becoming more variable as they approach the final layer. This trend experiences an abrupt disruption in layers 3 and 4, where all components exhibit a significant high value for the median of their \(r_{f}\) values. In consequence, as shown in Table 2, the third and fourth layers are lightly pruned for all components, except for the V component, in which a medium pruned is applied to the third layer. In contrast, layers from 5 to 7 are the less relevant for each component. Additionally, the Intermediate component is the most meaningful one. As a result, none of its layers is heavily pruned. In contrast, in the case of BERT Large, the median of the \(r_{f}\) values across the layers does not exhibit a consistent growth pattern. As observed, between layers 3 and 11, all components show increasing values, contributing more information over time. However, there are significant drops in the median of \(r_{f}\) for the Q, K, and V components between layers 12 and 13, as well as between layers 17 and 18. During these same layers, the Intermediate component experiences local maxima.