## 4 Materials and Methods
### Modeling and Evaluation
#### 4.4.1 Pre-Training BERTuncased  
In its pre-training phase, the BERTuncased model processes extensive textual data, notably from BookCorpus and Wikipedia, amounting to roughly 250 million words. This corpus undergoes an analysis through three Input Embeddings (_E_) for Next-Sentence  
Figure 8: Demonstration of DF traversal in the DOM for the search for embedded textual content. Classification (NSC) and MLM. In NLP, token embeddings methodically segment text into units corresponding to individual words. Positional embeddings assign each token a unique position, integrating this into the embedding layer for improved contextual understanding. Furthermore, segment embeddings are employed to discern between different text sequences, an essential aspect for accurately processing and interpreting linguistic data. To comprehend the mechanism by which \(\text{BERT}_{\text{uncased}}\) derives weights from \(E\), an examination of the components constituting the basic BERT architecture is delineated in the subsequent enumeration:  
1. _Concatenation with Positional Encoding_: \(E\) undergoes integration with the Positional Encoding (\(PE\)) layer, thereby infusing information regarding the sequential positioning of each token, resulting in \(E^{{}^{\prime}}=E+PE\). In this formulation, \(E^{{}^{\prime}}\) symbolizes the resultant sequences post-integration, embodying a composite of the tokens' semantic information and their respective positions within the sequence. 2. _Multi-Head Attention_ (\(MHA\)): \(E^{{}^{\prime}}\) is input into an attention layer for syntactic and semantic analysis of sequences, capturing the language's context and complexity. This is facilitated by the Multi-Head Attention (\(MHA\)), using matrix-weight tuples. Query matrices \(Q\) identify focus tokens in \(E^{{}^{\prime}}\), while key matrices \(K\) cover all tokens, crucial for computing attention weights and enabling query comparison. Value matrices (\(V\)) aggregate outputs from \(Q\) and \(K\) interactions. Each of the \(h_{i=1}^{12}\) heads applies dot-product attention to \(Q\), \(K\), and \(V\), generating scores transformed into probabilities via a softmax function scaled by \(\sqrt{d_{K}}\) (dimension of \(K\)). This scaling stabilizes the softmax function during weight re-calibration in the \(W^{Q}\), \(W^{K}\), and \(W^{V}\) matrices, preventing gradient reduction. See Equation (2) for more details. \[\text{attention}(Q,K,V)=\text{softmax}\Big{(}\frac{QK^{T}}{\sqrt{d_{K}}}\Big{)}V \tag{1}\]  
Consequently, each head \(h_{i=1}^{12}\) computes and concatenates the attention results \(h_{i=1}^{12}=\text{attention}(Q_{i},K_{i},V_{i})\), as detailed in Equation (2). Figure 9: Schematic of architecture for classifying categories \(MS\), \(M\), \(S\), and \(B\), utilizing a pre-trained BERT model for subsequent fine-tuning. \[MHA(Q,K,V)=\text{concatenation}(h_{1},\ldots,h_{12})W^{O},\] (2) where \(W^{O}\) is the weight matrix responsible for rescaling the outputs of the concatenation layer towards the subsequent stages of the process. 3. _Add & Norm_: the outputs of the \(MHA\) are subjected to a residual connection and normalization prior to entering the Feed Forward (FF) layer. This is integral in any Artificial Neural Network (ANN) to ensure that weights generated during training maintain their significance. This phase emphasizes two crucial processes: addition and normalization. Addition is crucial in mitigating gradient vanishing, as it preserves input weight information, denoted as \(I\), across the network. This preservation is maintained irrespective of subsequent layer transformations \(F(I)\), achieved by summing the input and transformed output, resulting in the residual output \(R_{O}=I+F(I)\). Subsequently, the role of normalization is to standardize \(R_{O}\). This standardization is accomplished using the mean-variance normalization method, detailed in Equation (3).