# A Green AI Methodology Based on Persistent Homology for Compressing BERT
## 2 Previous Works
### BERT Model Pruning Methods  
As mentioned earlier, to the best of our knowledge, there are no pruning methods for LLMs based on persistent homology. However, there are compression algorithms for the BERT architecture that provide competitive results [21]. These methods can be divided into structured and unstructured approaches [22].  
Among the structured methods, some of them focus on pruning attention heads exclusively. In [23], a regularization method for BERT is proposed using reinforcement learning to prune attention heads. For text classification, a compression method based on adaptive truncation is presented in [24]. In [25], a forward and backward pass is used to calculate gradients, which are used as an importance score for each attention layer. Ref. [26] suggests constructing a loss function that minimizes both classification error and the number of heads used, pruning unproductive ones while maintaining performance. Ref. [27] presents a technique with three stages: in the first, \(n\) pruning strategies are generated with the same pruning ratio. In stage II, \(n\) candidates from the training set are evaluated, and after all iterations, the one that performs the best on a subnetwork is chosen as the best candidate. This best candidate undergoes fine-tuning to obtain a good subnetwork. In [28], a new sentence-level feature alignment loss distillation mechanism for BERT compression is introduced. It is guided by a mixture of experts to capture contextual semantic knowledge from the teacher model to the student model while reducing its parameters. In [29], a dynamic structure pruning method based on differentiable search and recursive knowledge distillation technique called DDK is presented. It is focused on pruning all feed-forward and self-attention layers. Additionally, a recursive knowledge distillation method is presented, designed to extract the most important features from the intermediate layers. Another uncertainty-driven knowledge distillation technique is presented in [30]. The uncertainty modeling guides the training process effectively, especially when there is a large gap in network performance between pretrained language models and compressed ones. In [31], a Layer-wise Adaptive Distillation (LAD) method is proposed for model compression. Specifically, an iterative aggregation mechanism is designed to distill layer-wise internal knowledge from the teacher model to the student model. Finally, in [32], a novel approach to compress BERT, called You Only Compress Once-BERT (YOCO-BERT), is introduced. YOCO-BERT constructs a search space with all the possible configurations for BERT model. Then, by using an stochastic nature gradient optimization method, an optimal candidate architecture is generated.  
Among the unstructured methods, ref. [33] uses unstructured magnitude pruning to find subnetworks with sparsity levels between 40% and 90%. It concludes that those with 70% sparsity, using the masked language modeling task, are universal, and can solve other tasks without losing accuracy. Ref. [34] proposes a weight pruning algorithm that integrates reweighted L1 minimization with a proximal algorithm. More information on pruning algorithms for LLMs is available in [21].  
Quantization algorithms have also been proposed, such as [35], which uses second-order Hessian matrix information for quantizing BERT models to low precision. Hardware-based techniques have also been proposed [36]. In [37], a sensitivity-aware mixed precision quantization method called SensiMix is proposed, which applies a 1-bit quantization to insensitive parts of the model. Finally, there are knowledge distillation-based algorithms,like the one presented in [30], which is based on parameter retention and feed forward network parameter distillation.