## 5 Performance Analysis and Results
### Quantitative Assessment of User Satisfaction  
The effectiveness and usability of the virtual assistant were evaluated using user satisfaction as a critical metric. The understanding of queries by the VA, the relevance and quality of responses, and the overall user experience were primarily centered on user feedback. A Likert scale ranging from 1 to 5 was utilized to quantify various aspects, while additional binary assessments were implemented to capture specific response attributes, including accuracy and hallucinations. The results from user testing were found to indicate a generally favorable impression, with an average satisfaction rating of 3.575 recorded across all responses, thereby underscoring the VA's effectiveness in meeting most users' expectations, as shown in Figure 7:  
The syntactic and grammatical quality received the highest scores, with an average of 4.425 noted from the testing. The score indicates that clarity and professionalism were perceived in the responses, which is crucial for maintaining trust in automated systems. Many users noted that the reactions were stylistically correct and well structured, contributing positively to the perceived reliability of the VA. Despite this, it was observed that responses were occasionally rated highly by some users based solely on grammatical quality, with minor factual inaccuracies or hallucinations potentially being overlooked.  
Figure 6: Hallucination Occurrence by Context Relevance.  
The ability of the VA to comprehend and interpret queries was rated at 4.225, indicating that users generally felt that the system understood their questions adequately. The importance of this metric is highlighted, as it reflects the system's performance in parsing user intent, which is considered a key component for any language model-based assistant. The average score of 3.825 for response relevance was observed, indicating occasional mismatches between user queries and the answers provided by the VA. Minor misinterpretations of the context were often responsible for these mismatches, resulting in slight deviations from the intended answer, particularly in questions that involved specific institutional policies or requirements.  
The overall user experience, while positive, was found to have room for improvement. The final satisfaction metric of 3.575 indicated that users perceived the VA as a helpful tool; however, limitations in response accuracy were also encountered. It has been observed that incorrect or hallucinated responses, while infrequent, have had an impact on user confidence in the reliability of the VA. Users have been required to cross-reference answers with additional sources to ensure correctness. The feedback suggested that in contexts where hallucinations occurred, users would appreciate a more unambiguous indication of confidence levels or disclaimers to help mitigate potential misinformation risks.  
This quantitative assessment provided valuable insights into the strengths and limitations of the VA. Although linguistic and syntactic quality received high ratings, it is recognized that improvements in context retrieval and response accuracy are required for broader user satisfaction. By refining these aspects, the VA may achieve a more dependable tool in an academic setting, resulting in increased user confidence in its responses and enhanced overall user experience.