# A Green AI Methodology Based on Persistent Homology for Compressing BERT  
[PERSON]  
1 Department of Computer Science and Artificial Intelligence, University of Granada, 18071 Granada, Spain; j.m.  
In this article, Persistent BERT Compression and Explainability (PBCE) is proposed, a Green AI methodology based on homology theory to compress the BERT model and give insights about the importance of its neurons. BERT has been chosen as the reference model, given the fact that is one of the foundational models that laid the groundwork for LLM encoders. Several subsequent BERT-based encoder models, such as RoBERTa [5], DistilBERT [6], ALBERT [7], or TinyBERT [8], have been developed to address specific limitations of the original. PBCE is focused on the internal representation of the neural network, considering the individual role of each neuron when the model makes an inference. In particular, PBCE uses zero-dimensional persistent homology, extracting the topological characteristics of the neurons' outputs and providing an assessment of their importance within the network. This way, it is feasible to identify neurons that contribute with more information to the overall computation of the network and pruning those that contribute less. As a consequence, the proposed technique becomes an effective method for compressing BERT.  
To the best of our knowledge, this is the first approach that uses persistent homology to compress and give insights of the importance of the units of the BERT model. To measure the effectiveness of PBCE, an extensive experimentation has been designed based on natural language processing tasks proposed in General Language Understanding Evaluation (GLUE) benchmark [9], achieving results that outperform other state-of-the-art techniques for BERT compression. The main purpose of the experimentation is to provide an answer to the following research questions (RQ):  
* How effective and robust is persistent homology at measuring the importance of neurons in a transformer encoder model such as BERT?
* Is it feasible to propose a practical methodology using persistent homology for simplifying BERT-based models?
* Can persistent homology be employed to enhance the explainability of language models like BERT?  
The main contributions of the article can be summarized as follows:  
* A methodology has been developed for compressing the BERT model by analyzing the topological features of the outputs of each neuron. This can be understood as explainability in terms of the individual role of each neuron.
* This methodology is applied to two versions of the BERT model, interpreting the topological characteristics as a tool to assess the importance of neurons, simplifying those that contribute less information, and generating a pruned version of the BERT model.
* The performance of the simplified models has been evaluated using the GLUE Benchmark and compared the results with other state-of-the-art compression techniques, demonstrating the effectiveness of PBCE for model explicability and compression.  
The rest of this paper is structured as follows: In Section 2, the state of the art of different approaches using persistent homology and deep learning models to solve problems and BERT compression techniques is introduced. In Section 3, PBCE is described. In Section 4 the methodology is experimentally analyzed and discussed. Section 5 highlights the conclusions.