# Evaluating Retrieval-Augmented Generation Models for Financial Report Question and Answering
## 5 Evaluation and Discussion of Results
### Additional Results and Discussion  
Table 7 presents the evaluation results for each bank, independent of the model configuration. The main difference was observed between Barclays (290 of 450 points) and HSBC (338 of 450 points). Barclays, with an average score of 3.2, had a notably low score for \"Context Relevance\". A comparison of the half-yearly and quarterly reports of Barclays and HSBC reveals that they use completely different layouts. Barclays reports are in landscape mode and contain a large amount of data displayed in tables and various types of charts \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**No.**} & \multirow{2}{*}{**Embedding Model**} & \multirow{2}{*}{**LLM**} & \multirow{2}{*}{**Points**} & **Metric 1:** & **Metric 2:** & **Metric 3:** & \multirow{2}{*}{**Total**} \\  & & & **Context** & **Answer** & **Answer** & **Answer** & \\  & & & **Relevance** & **Faithfulness** & **Relevance** & \\ \hline \multirow{2}{*}{1} & \multirow{2}{*}{OpenAI ADA} & \multirow{2}{*}{OpenAI GPT-4o} & Total Points & 175 & **189** & **188** & **552** \\ \cline{3-8}  & & & **Average Points** & **3.5** & **3.78** & **3.76** & **3.68** \\ \hline \multirow{2}{*}{2} & \multirow{2}{*}{\begin{tabular}{c} OpenAI ADA \\ (consistent with \\ baseline) \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} Google Gemini \\ 1.5 Pro \\ \end{tabular} } & Total Points & **176** & 185 & 182 & 543 \\ \cline{3-8}  & & & **Average Points** & **3.52** & 3.7 & 3.64 & 3.62 \\ \hline \multirow{2}{*}{3} & \multirow{2}{*}{\
\begin{tabular}{c} MiniLM \\ Embedder \\ \end{tabular} } & \multirow{2}{*}{
\begin{tabular}{c} OpenAI GPT-4o \\ (consistent with \\ baseline) \\ \end{tabular} } & Total Points & 137 & 155 & 146 & 438 \\ \cline{3-8}  & & & **Average Points** & 2.74 & 3.1 & 2.92 & 2.92 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Evaluation results: three model configurations (best results are indicated in bold). (e.g., line charts and bar charts) with minimal written text. On the contrary, HSBC's reports are text-heavy with well-explained tables and minimal charts. This layout makes it easier for retrievers to gather the relevant context. Consequently, the HSBC reports are more suitable for the RAG system. Table 8 presents the results for each of the seven standards and three individual questions. The highest scores were obtained on individual questions. This indicates that when specific questions and wording align with the report, the RAG system performs well across all three metrics, thereby providing accurate answers. The qualitative questions also received higher average scores than the quantitative questions. It was found that when multiple similar numbers appeared in a report, the RAG sometimes had difficulties handling them accurately. For example, question 4 asked for the total assets held by the bank at the end of H1 2023. Banks typically report various types of assets, such as risk-weighted assets, customer assets, or high-quality liquid assets. If the RAG system selected the wrong number, it received a low score. However, minimal hallucination was observed; instead, the primary issue was selecting an incorrect but actually appearing number. On the other hand, for qualitative questions such as market trends, the RAG system often produced at least a partially correct answer, which resulted in higher scores. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Bank** & \begin{tabular}{c} **Metric 1: Context** \\ **Relevance** \\ \end{tabular} & \
\begin{tabular}{c} **Metric 2: Answer** \\ **Faithfulness** \\ \end{tabular} &
\begin{tabular}{c} **Metric 3: Answer** \\ **Relevance** \\ \end{tabular} & **Total** \\ \hline Banco Santander & 96.0 & 97.0 & 99.0 & 292.0 \\ \hline CS & 104.0 & 97.0 & 97.0 & 298.0 \\ \hline Barclays & 81.0 & 107.0 & 102.0 & 290.0 \\ \hline Credit Agricole & **106.0** & 107.0 & 102.0 & 315.0 \\ \hline HSBC & 101.0 & **121.0** & **116.0** & **338.0** \\ \hline \hline \end{tabular}
\end{table}