## 3 Materials, Methods, and Findings
### Model Implementation  
The question encoder part of the RAG was replaced with a powerful model such as BERT. A learning rate scheduler is used to dynamically adjust the learning rate according to epochs. The early stop mechanism is enabled to prevent the model from overlearning. The embedding representations are enriched by adding self-attention or multi-head attention mechanisms. This allowed us to better capture the contextual meaning of the RAG embeddings. The learning of the contextual relationships is improved by processing the RAG-derived embeddings with a transformer encoder. This provides a structure that further enriches the contextual embedding obtained from the RAG model. In this structure, the embeddings from the RAG are passed through a fully connected layer to the enriched contextual representations. These representations are used in the classification layer to obtain the results. The RAG-ECE model architecture is shown in Figure 21.  
Figure 20: RAG-ECE confusion matrix. The mathematical implementation of the proposed RAG-ECE model is calculated by Equations (1)-(7) [21; 22; 23]. The raw input data \(x\) are preprocessed to create an initial feature representation \(h_{x}\):  
\[h_{x}\ =\ Preprocess(x) \tag{1}\]  
where \(Preprocess(\cdot)\) includes tokenization, normalization, or other preprocessing tasks. The embedding model transforms the preprocessed input \(h_{x}\) into a latent space representation \(h_{embed}\):  
\[h_{embed}\ =\ E(h_{x}) \tag{2}\]  
where EEE represents the embedding model. Relevant information chunks \(k_{i}\) are retrieved from the vector store based on the similarity between the embedding representation \(h_{embed}\) and the stored vector representations D = {d1,d2,...,dn}. The probability distribution over the chunks is computed as:  
\[P(k_{i}|h_{embed})\ =\ softmax(h_{embed}.d_{i}) \tag{3}\]  
The final retrieved context \(k\) is a weighted sum of the retrieved chunks:  
\[k\ =\ \sum_{i}P(k_{i}|h_{embed}).d_{i} \tag{4}\]  
The retrieved embedding \(k\) is passed through the Enhanced Contextual Embedding module, consisting of fully connected layers and a \(RELU\) activation function. First fully connected layer:  
\[h_{ECE}\ =\ RELU\ (w_{1}k+b_{1}) \tag{5}\]  
Second fully connected layer:  
\[h_{ECE}\ =\ w_{2}h_{ECE}+b_{2} \tag{6}\]  
Here, \(w_{1},w_{2}\) are weight matrices, and \(b_{1},\ b_{2}\) are bias terms. The enhanced embedding \(h_{ECE}\) is passed to the large language model (\(LLM\)) for the final output generation:  
\[y\ =\ LLM(h_{ECE}) \tag{7}\]  
Figure 21: RAG-ECE model architecture. Contextual Embedding: RAG-ECE enhances the contextual representations of the texts by utilizing the retrieval capability of the RAG model, adding more fully connected layers on top. Multi-Head Attention: A multi-head attention mechanism has been implemented on the contextual representations to make the representations more meaningful by capturing different contextual relations in texts through different heads. Transformer Layers: The architecture adds some transformer encoder layers that enable the model to learn deep semantics. These layers process the contextual information in the text representations more powerfully and feed it to the classifier. Hyperparameter optimization to improve model performance has played a critical role in the text classification process. Automated hyperparameter optimization tools, such as Optuna, were used to determine the parameters that would give the best performance of the model. The optimized parameters are as follows:  
Hidden Layer Size: The number of neurons in the fully connected layers is optimized. Learning Rate: The learning rate used during the training of the model is optimized so that the model learns faster and more efficiently. Batch Size: By optimizing the batch size, the size of the data pieces used during the training is adjusted. The optimal hyperparameters for each model were determined experimentally and evaluated according to the performance on the validation set.