# Evaluating Retrieval-Augmented Generation Models for Financial Report Question and Answering
## 3 Research Design
### Process Steps of Design Science Research
#### 3.1.3 Development  
After defining the problem and identifying the artifact type, as per [PERSON] and [PERSON] [29], the Development phase should concentrate on designing and creating an artifact that offers a solution to the defined problem. The sub-questions one and two (see Table 1), outlined in Section 1.2., are integral to this phase. \begin{table}
\begin{tabular}{c l l} \hline \hline
**No.** & **Sub-Questions** & **Activities** \\ \hline \multirow{3}{*}{1.} & How can the retrieval component of RAG & Data collection through literature \\  & be set up to extract relevant information & review \\  & from half-yearly or quarterly reports? & Artifact development \\ \hline \multirow{3}{*}{2.} & What strategies can be employed to ensure & \\  & that the generation component of RAG & Data collection through literature \\  & produces answers to questions posed about & review \\  & half-yearly or quarterly reports? & Artifact development \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Sub-questions for the Development phase. In this phase, the focus is on developing and implementing an RAG system. Establishing an RAG system requires the definition and selection of certain components. Figure 1 shows the simplified architecture. The flow is outlined based on the following 9 steps (identified by the circled number in Figure 1) based on [6]:  
1. Begin by gathering essential reports for the RAG system's operation, specifically targeting half-yearly and quarterly reports from banks. Further insights into the data collection procedures are provided in Section 3.2. 2. Break down the collected data into manageable chunks to streamline information retrieval and improve efficiency by avoiding the processing of entire documents. This segmentation ensures that each data chunk remains focused on a specific topic, which increases the likelihood of retrieving relevant information for user queries. Subsequently, these segmented data are transformed into vector representations known as embeddings, which capture the semantic essence of the text. 3. The resulting embeddings are stored in a dedicated vector database to facilitate the efficient retrieval of pertinent information, transcending traditional word-to-word comparison methods. 4. The user query formulation is initiated. 5. Upon entry into the system, the user query is converted to an embedding or vector representation. To ensure consistency, the same model is used for both document and query embeddings. 6. Use the transformed query to search the vector database and perform comparisons with document embeddings. 7. The most relevant text chunks are retrieved, and a contextual framework is established to address the user query. 8. The retrieved text chunks are integrated with the original user query to provide a unified prompt for LLM. 9. The unified prompt, which comprises the retrieved text chunks and the original user query, is sent to the LLM. The LLM utilizes its advanced natural language processing capabilities and extensive knowledge to generate coherent responses tailored to address the user's queries effectively, leveraging the additional context provided by the text chunks to enhance the accuracy and depth of the responses. Therefore, for the development of a newly built RAG system, the essential components include an embedding model, a vector database, an LLM, and a comprehensive orchestra  
Figure 1: RAG system according to [6]. tion tool to manage the entire system seamlessly. The following tools were identified as potential setups:  
* Embedding Model: MiniLMEmbedder [30] and BedRockEmbeddings [31];
* Vector Database: FAISS [32] and Chroma [33];
* LLM: GPT-4o [34], Llama 3 [35], and Gemini 1.5 Pro [36];
* Orchestrator: LangChain [37] and LlamaIndex [38]. Furthermore, there exists an alternative approach involving the use of a pre-built RAG system that can be readily deployed. One such system is the Verba RAG by Weaviate.