## 4 User Testing Methodology
### Detailed Scoring  
The response patterns of the VA were further illustrated through the meticulous application of specific scoring methods to assess various critical aspects of the responses. The foundational accuracy metric was established, with each response being evaluated through a binary score that determined whether the VA provided factually correct information. Responses were designated as \"Yes\" for accuracy when correct data relevant to the query was contained or \"No\" when errors were identified. The binary approach was offered as a straightforward yet essential measure of the VA's reliability in delivering accurate information, regarded as a core requirement for an educational support system. Furthermore, Hallucination Detection was implemented as a binary score, with responses that included content deviating from the context or intent of the question being flagged. Instances of hallucinations, characterized by the VA producing information without a basis in the available data, were identified. This scoring metric addressed an essential concern in AI-driven applications. The detection and mitigation of hallucinations are considered vital to ensuring that the VA is maintained as a dependable source of information and is not inadvertently used to mislead users. Another crucial component, Logical Errors, was assessed through a binary check to identify coherence issues within the responses. The scoring aimed to detect cases in which the VA's responses exhibited logical fallacies or inconsistencies with the posed questions, thereby undermining the overall coherence and reliability of the response. Logical consistency in responses is recognized as essential for maintaining user trust, especially in the context of complex or ambiguous queries. Relevance and Quality were assessed on a Likert scale from 1 to 5 to complement these binary scores. This approach permitted a more nuanced evaluation of the alignment between the VA's responses and the user's query intent. This scale considered the contextual appropriateness and syntactic quality of each response, ensuring that the VA addressed the content of the question. At the same time, clarity and professionalism were maintained in its presentation. High scores in this area indicated that the VA's answers were well suited to the inquiry and professionally constructed, contributing positively to user experience. Finally, the overall user experience was evaluated using a Likert scale, and participants' subjective impressions of their interaction with the VA were captured. The scoring was conducted considering ease of use, response clarity, and overall satisfaction, providing insights into user-centered aspects of the VA's performance. The holistic user experience was focused on, leading to the identification of areas in which the VA excelled at user engagement, along with any friction points that could potentially impact user adoption. In addition to the structured scoring system, the analysis of examples of both positive and negative responses allowed for more profound insights into the VA's strengths and areas needing improvement. Illustrative samples of these responses were provided in Tables 2 and 3 from the thesis, with specific user queries paired with the VA's generated answers. These tables facilitated the documentation of cases where the VA performed effectively and instances in which contextual relevance or accuracy struggles were observed. A positive response, as illustrated in Table 2, was featured about a commonly asked administrative question. The VA successfully retrieved accurate, contextually relevant information and presented it with proper syntax and grammar, resulting in high scores in relevance and quality. This sample highlighted the potential of the VA for handling straightforward queries reliably.