## 3 Materials and Methods
### Evaluation: RAGAs Framework  
We used the RAGAs framework [19], which is a framework that focuses on evaluating the retrieval and generation capabilities of RAG systems, to evaluate the performance of the proposed system. The evaluation of each component of the RAG pipeline can be divided into two parts: answer generation and document retrieval.  
The generation process shown in Figure 8 comprises two metrics: faithfulness, which evaluates the relevance between the retrieved documents and generated answers, and answer relevance, which evaluates the relevance of the generated answers to the questions. In the search process, the documents retrieved for a question are evaluated based on context precision and recall.  
The mathematical formulas for the evaluation metrics are summarized in Table 1. This table presents the formulas used to assess answer generation and document retrieval in the RAGAs framework.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Metric** & **Equation** \\ \hline Faithfulness & Faithfulness score \(=\frac{\text{[Number of claims in the answer that can be inferred from the given context]}}{\text{[Total number of claims in the answer]}}\) \\ \hline Answer Relevancy & answer relevancy \(=\frac{1}{N}\sum\limits_{i=1}^{N}\text{cos}\big{(}E_{\mathcal{G}},E_{ \mathcal{G}}\big{)}\) \\ Answer Relevancy & answer relevancy \(=\frac{1}{N}\sum\limits_{i=1}^{N}\frac{E_{\mathcal{G}}\cdot E_{ \mathcal{G}}}{\|E_{\mathcal{G}}\|\|E_{\mathcal{G}}\|\|E_{\mathcal{G}}\|}\) \\ \hline Context Precision & Context Precision@K \(=\frac{\sum\limits_{i=1}^{N}(\text{Precision@K\times\cap\cap}\text{k})}{ \text{Total number of relevant items in the top K results}}\) \\ Precision@K \(=\frac{\text{true positives@K}+\text{false positives@K}}{\text{(true positives@K+ false positives@K)}}\) \\ \hline Context Recall & context recall \(=\frac{\text{[Ground Truth] sentences that can be attributed to context]}}{\text{[Number of sentences in {Ground Truth]}}}\) \\ \hline \hline \end{tabular}
\begin{tabular}{c} \(E_{\mathcal{G}}\) is the embedding of generated question \(i\), \(N\) is the total number of generated questions; \(E_{\mathcal{G}}\) is the embedding value for the original question, and k is the total number of chunks.  
\end{table}
Table 1: RAGAs evaluation metrics.  
Figure 8: RAGAs evaluation.  
The faithfulness metric evaluates the consistency of the generated answers for providing relevant information based on the given context. It is calculated by comparing the generated answer with the context, with a higher probability indicating a more reliable answer. To calculate faithfulness, a set of assertions in the generated answer is first identified, and then, each assertion is crosschecked against the given context to determine whether the assertions can be inferred from the context.  
Answer relevance indicates the relevance of the generated answer to an initially posed question. It evaluates the extent to which the answer meets the requirements of the question, with a high score indicating a complete and clear answer to a given question without redundant or unnecessary information. To calculate this, we used the LLM to generate multiple appropriate questions for the generated answers and evaluated their relevance by measuring the average cosine similarity between the generated and original questions.  
Context precision indicates the percentage of retrieved documents containing content relevant to a question. It is used to evaluate the accuracy with which a search system curates and presents relevant documents to users. Context recall comprehensively evaluates whether the retrieved documents contain the information required to formulate the answer to a given question. It evaluates the performance of the search system during document retrieval by assessing whether the document contains sufficient background information and the correct answer to the question.