# A Green AI Methodology Based on Persistent Homology for Compressing BERT
## 3 Our Proposal
### An Intuitive Geometric Description of Persistent Homology Applied to LLM Explanations  
Homology theory is a branch of algebraic topology increasingly used in data science. In particular, persistent homology is a powerful tool for studying patterns in data. Its mathematical foundation, summarized in Appendix A, is deep and complex with a significant algebraic burden. However, it can be given a much more intuitive geometric interpretation, which will be explained below. In Figure 3, a complete example of data processing in BERT can be found. Starting with a set of elements from a corpus, after being tokenized, they are fed into the network. The output of any neuron in a hidden layer, which serves as input for the next layers, consists of vectors that, in an abstract sense, could be represented as points in a hyperspace. The geometric distribution of these points can provide meaningful information about the behavior and role that the corresponding neuron plays within the data flow of the network. Zero-dimensional persistent homology helps explain this role. Figure 3: Representation of information through the BERT model and subsequent extraction of values from intermediate dense layers. The process begins with the processing of a set of elements from a corpus. Once the input of \(N\) sentences with length \(M\) is constructed, it is fed into the neural network. In the lower right part of the image, the output of any dense layer is represented. The output is a three-dimensional matrix: the number of sentences in the input (\(N\)), the length of those tokenized sentences (\(M\)), and the number of hidden neurons in that layer. This information, in the form of an n-dimensional matrix, is taken to represent the information associated with each neuron. To analyze it, persistent homology is used. As can be seen in Figure 3, each BERT layer generates multidimensional vectors of size \(N\times M\times NHU\), with \(N\) being the number of output rows, \(M\) the number of tokens in each sequence, and \(NHU\) the number of hidden units. These vectors represent the internal state of texts used as inputs. For the sake of clarity, a simple two dimensional example is introduced in Figure 4. Two plots at three different time moments can be seen in that figure. The plot on the left depicts some vectors generated by any layer (as explained earlier), which are going to be converted into the centers of disks. The disks will evolve, growing uniformly with a common radius. The value of the radius is marked by the red line on the right (Birth-Death diagram). Notice that this red line will move upwards on the Y-axis of the graph on the right. Figure 4: Application of persistent homology on the output of a neuron at three specific moments. On the left, it can be seen that each of the points comprising the output becomes the center of a disk whose radius grows uniformly for all points. On the right, the Birth–Death diagram is represented for persistent homology of dimension zero. Each blue point corresponds to the disappearance of a connected component after collapsing with another. The last moment depicted in the figure represents the point at which the value of \(r\) is reached for which all connected components first merge. This value is called \(r_{f}\), and is crucial in the proposed methodology because it provides information about the importance of neurons based on their output within the neural network’s data flow. Since PBCE is based on zero-dimensional persistent homology, each output vector becomes a singleton connected component and all originate at time zero (value 0 on the Birth-Death diagram). In consequence, there are as many connected components as output vectors. As the radius of the disks grows, the connected components eventually collapse. When two connected components touch, they merge becoming a single area (a single component), and a blue point is marked on the Birth-Death plot at the radius value that led to the merger. This point on the Birth-Death plot implies the death of a connected component.