# Retrieval-Augmented Generation (RAG) Chatbots for Education: A Survey of Applications
## 3 Results
### Performed Evaluation  
The fact that the performance of RAG-based chatbots depends on multiple factors, including the choice of retrieval model, the large language model, the considered corpus, the prompt formulation, and the tuning of all these components [61], their evaluation is of primary importance, as even high-quality components do not guarantee satisfactory results. The problem is that there is no widely accepted standard for the evaluation of RAG-based chatbots. As they have several distinct quality facets, the concepts for their evaluation are adopted from different areas of research:  
* Because of the retrieval component, we have examples of evaluation based on classic information retrieval measures [62], such as Precision, Recall, F1 Score, Accuracy, Sensitivity, and Specificity; these are instrumental in comparing the performance of different approaches that can be used to generate vector representations of the queries and responses--for instance, F1 Score reported in [25] (p. 5697) ranges from 0.35 for GloVe [63] to 0.67 for TF-IDF [64];
* Because of the LLM component, we have examples of evaluation based on LLM benchmarks, such as AGIEval [65], measuring the ability to pass human-centric standardized exams; this is instrumental in comparing the performance of different LLMs that can be used as the base of the RAG chatbots--for instance, AGIEval measurements reported in [55] range from 21.8 for Falcon-7B [66] to 54.2 for LLaMa2-70B [67];
* Because chatbots are made for conversation, we have examples of evaluation based on conversational quality measures, in particular based on [PERSON]'s Maxims of Conversation [68], such as Correctness (assessing whether the chatbot's answer is factually correct), Completeness (assessing whether the answer includes all relevant facts and information), and Communication (assessing whether the answer is clear and concise) [32], or (in [PERSON] et al. 's interpretation) Information Quantity (assessing whether the provided amount of information is neither too much nor too little), Relevance (assessing whether the answer is pertinent to the ongoing topic or the context of the conversation), Clarity (assessing whether the answer is clear, concise, and in order), and Credibility (assessing whether the recipient trusts the chatbot) [53]; these could be used for, e.g., comparing the users' trust in the answers generated with the help of different LLMs that can power the RAG chatbots (e.g., [PERSON] et al. compared the trust level between two versions of GPT: 3.5 and 4.0 [69], and reported 5.50 for answers generated by GPT 3.5 and of 5.59 for answers generated by GPT 4.0 [53]), but also the  
Figure 4: Used large language models. change in the users' trust due to incorporating RAG--interestingly and unexpectedly, the same source reports a decrease in the average trust level for answers generated by both GPT 3.5 (from 5.98 to 5.50) and GPT 4.0 (from 6.34 to 5.59) after combining them with the content extracted from lecture slides and their transcripts [53];
* Because chatbots are instances of generative Natural Language Processing, we have examples of computer-generated text quality measures such as the similarity to human-written text (measured with metrics such as K-F1++ [70], BERIscore [71], BLEURT [72], or ROUGE [73]) [28; 46], text readability (measured with, e.g., Coleman-Liau [74] and Flesch-Kincaid [75] metrics) [44], and human-assessed text quality aspects such as Fluency, Coherence, Relevance, Context Understanding, and Overall Quality [17]; these, again, can be instrumental in comparing the performance of different LLMs that can be used as the base of the RAG chatbots--for instance, [PERSON] et al.