# A Green AI Methodology Based on Persistent Homology for Compressing BERT
## 2 Previous Works
### Brief Description of the BERT Model  
BERT, an acronym for \"Bidirectional Encoder Representations from Transformers\", Ref. [1] is a language model developed by Google that has revolutionized natural language processing. Its primary innovation lies in its ability to comprehend a word's context within a text by analyzing all surrounding words, both to the left and right. Unlike previous language models, which were unidirectional and predicted words based on previous ones, BERT takes the entire context into account.  
BERT is built upon the Transformer architecture, a deep neural network autoencoder. The Transformer architecture employs multi-head attention to process sequences. Before the network processes the information, the texts are tokenized, meaning texts are divided into a set of tokens that represent the fundamental semantic content of the word and are translated into their numerical position in the model's vocabulary. Additionally, special tokens are added, such as [CLS], marking the start of a sentence; [SEP], marking the end, and [PAD], signifying the end of padding. Some research papers from the literature, such as [18], show that the [CLS] token, besides marking the beginning of a sequence, provides an aggregated representation of the input. In fact, it is used in text classification tasks and sentiment analysis. Figure 1 shows an example of how sequences are generated to be a valid input for BERT model.  
To pre-train and fine-tune BERT, an extensive amount of text was used. During the training process, BERT learns to predict hidden words in complete sentences, enabling it to better understand how words relate to each other in a given context.  
This pre-trained model can be fine-tuned for specific tasks such as text classification, entity tagging, machine translation, and more. Due to its context-awareness, BERT has outperformed many records in various natural language processing tasks and has become an essential tool in this field.  
Each BERT Layer, as part of the Encoder of the Transformers architecture, consists of three components: the Multi-Head Attention layer, comprises the query (Q), key (K), and value (V) matrices, and Attention Output; the Intermediate layer; and the Output layer. Figure 2 shows the complete representation of the BERT architecture. Additionally, more detailed information can be found in [1,4].  
Figure 1: Usage of BERT involves taking a sentence, adding the special tokens [CLS] and [SEP], tokenizing the words, and using these tokens as input for the neural network.  
Figure 2: Representation of the BERT architecture. It is composed of an embedding module, followed by the Encoder part, which consists of \(N\) BERT Layers (12 or 24, depending on whether it is BERT Base or Large). Within it, three main components stand out: the Attention layer, the Intermediate layer, and the Output layer. After the Encoder, BERT has a Pooler layer.  
Let \(L\) be the number of layers, \(H\) the hidden size, and \(A\) the number of self-attention layers of the model. There are two implementations of the BERT model: the BERT Base Cased model [19] (\(L=12\), \(H=768\), \(A=12\), total number of parameters \(=110\) M, and the BERT Large Cased model [20] (\(L=24\), \(H=1024\), \(A=16\), total number of parameters \(=340\) M).