# Evaluating Retrieval-Augmented Generation Models for Financial Report Question and Answering
## 5 Evaluation and Discussion of Results
### Main Results and Discussion  
The primary outcome is the evaluation of the three model configurations (see Table 6). Our initial goal was to determine which model configuration performed best and to address research questions 3 and 4 from Section 1.2.  
\begin{table}
\begin{tabular}{p{85.4 pt} p{284.5 pt}} \hline \hline
**Scale** & **Definition** \\ \hline
1 & The retrieved context is irrelevant to the question, the generated response is inaccurate and inconsistent, and it fails to address the question. \\ \hline
2 & The retrieved context has limited relevance and contains several inaccuracies, the generated response exhibits notable inconsistencies, and it only partially addresses the given question. \\ \hline
3 & The retrieved context is somewhat relevant but has some inaccuracies, the generated response is generally consistent with minor errors, and it adequately addresses the question. \\ \hline
4 & The retrieved context is relevant and mostly accurate, the generated response is consistent with minor or no errors, and it effectively addresses the question. \\ \hline
5 & The retrieved context is highly relevant and accurate, the generated response is entirely faithful to the context with no errors, and it thoroughly addresses the question. \\ \hline \hline \end{tabular}
\end{table}
Table 5: Scale for evaluation.  
Model one (OpenAI ADA and OpenAI GPT-4o) achieved the highest overall score (552 of 750 points). With an average score of 3.7 out of 5, this model configuration falls between the 3- and 4-point definitions on the scale presented in Section 5.1. Thus, the retrieved context was generally relevant and mostly accurate, and the generated responses were consistent with minor errors to address the posed questions. The lowest scores were assigned to \"Context Relevance\", often due to retrieving over 10-15 chunks, many of which were not pertinent. However, even with irrelevant chunks, the LLM still provided mostly accurate answers. This highlights GPT-4o's robustness in handling excess information.  
Model two (OpenAI ADA and Gemini 1.5 Pro) received a slightly lower overall score. The \"Context Relevance\" score remained nearly stable due to using the same embedding model. However, the scores for \"Answer Faithfulness\" and \"Answer Relevance\" were slightly lower, reflecting the impact of using a different LLM. This observation aligns with benchmarks from the LLM Leaderboard of the Large Model Systems Organization [44], where GPT-4o was rated with 1287 points compared to Gemini 1.5 Pro with 1266 points. In addition, the Massive Multitask Language Understanding (MMLU) benchmark developed by [PERSON] et al. [45] obtained a score of 88.7% for GPT-4o and 81.9% for Gemini 1.5 Pro [46]. These benchmarks indicate a slight performance edge for GPT-4o, which was reflected in the evaluation results.  
Model three (MiniLM Embedder and OpenAI GPT-4) demonstrated a significantly lower overall score of 438 points, which was more than 20% lower than model one and lower than model two. Changing the embedding model from OpenAI's ADA to the MiniLM Embedder resulted in lower scores for all the metrics, which is not unexpected as the new model is much smaller than ADA. The \"Context Relevance\" score was considerably lower because much of the retrieved context was not relevant to the posed questions. We observed that this model configuration often retrieved fewer chunks than the previous models, which also affected \"Answer Faithfulness\" and \"Answer Relevance\". Without sufficient (quantity) and relevant (quality) context, the LLM struggled to generate accurate answers. This underscores the importance of using a high-quality embedding model to retrieve relevant information.  
In summary, to answer research questions three and four from Section 1.2, models one and two demonstrated a more accurate representation of the necessary information, while model three lagged behind. The evaluation approach and setup were effective for reliably evaluating and validating these experiments.