## 4 User Testing Methodology  
The functionality and reliability of a VA specifically developed to assist call center agents within an academic setting were evaluated through user testing in this study. The design of user testing was aimed at addressing two main aspects: first, the examination of the VA's ability to generate accurate and contextually relevant responses, and second, the identification of potential adverse effects of incorrect or misleading responses on end users. Given the importance of precise and timely information in educational institutions, the VA must provide dependable assistance to students and administrative staff without introducing confusion or inaccuracies. The validation of AI-driven applications, especially in contexts where the information provided may influence user actions and decisions, is critically supported by user testing. In this study, the effectiveness of the VA is assessed not only in terms of response accuracy but also in its ability to understand and incorporate context into responses, thereby minimizing errors. The significance of this requirement is underscored by the potential repercussions associated with incorrect responses in an educational setting. Consequently, user testing aims to identify areas in which the VA performs well and areas in which refinement may be required to meet the high standards of reliability expected within a university's administrative support framework. Two hypotheses were formulated to structure the evaluation. Hypothesis 1 posits that the VA can effectively understand and respond to user queries when contextual data is provided. The core expectation that the VA can utilize contextual information to enhance response accuracy and relevance is reflected in this hypothesis. The VA leveraged the RAG methodology to access and apply semantically relevant literature from its database, allowing for the generation of contextually appropriate responses that address user queries accurately. It is emphasized that the VA can process contextual data effectively and align responses with user intent. Hypothesis 2 proposes that the VA limits the potential for harm to end users by minimizing incorrect or misleading information. The risks associated with hallucinations or erroneous outputs, which are inherent in many large language models, are addressed by this hypothesis. Given the reliance on generated responses in a university's operational setting, assessing whether the VA can provide accurate responses without introducing information that could negatively affect user decisions is crucial. The relevance of this hypothesis is particularly noted in scenarios where inaccurate responsesmay misinform students or staff regarding critical processes, such as course registration deadlines or appeals procedures. The user testing methodology was carefully designed to align with these hypotheses, ensuring a structured and systematic evaluation of the VA's performance. A representative group of participants, consisting of students and administrative staff, was included in the testing process, and they interacted with the VA in simulated real-world scenarios. Feedback was gathered from a diverse participant base, allowing for reflection on the different types of inquiries that the VA might encounter in daily operations. To assess Hypothesis 1, various query types were incorporated into the testing framework, each designed to examine the VA's ability to retrieve and apply relevant contextual data effectively. Questions were submitted to the VA by participants in two modes: one mode allowed access to contextual data, while the other did not provide such data. The extent to which contextual information influenced the VA's performance was determined by comparing responses generated in these two modes. Each response was evaluated based on accuracy, relevance, and coherence, with particular attention given to the VA's ability to interpret and apply context effectively.