## 5 Performance Analysis and Results
### Limitations and Areas for Improvement  
Implementing the virtual assistant utilizing the RAG method in conjunction with the VERTEX AI platform demonstrated significant strengths; however, limitations were also highlighted, particularly in handling context-sensitive data and maintaining response accuracy across varied queries. The generalization issue was identified as one of the main challenges. Although effective in grounding responses, the RAG method occasionally struggled with generating accurate responses in instances where the input context was either highly specific or sparse. The tendency to generalize excessively is attributed to the reliance of the language model on the immediate context, which can sometimes narrow the scope of relevant responses.  
Figure 7: User Satisfaction Rating Distribution Across Metrics.  
Furthermore, obstacles were presented by overfitting and bias in specialized topics, as inherent biases from the training data were sometimes reflected in the model's responses. It was particularly evident in topics where neutral perspectives or diverse viewpoints were lacking in the context retrieved. The need for continuous model tuning and enhanced training data to mitigate these biases is underscored by these instances. Furthermore, the model's reliance on context limited the creativity and flexibility of responses, as highly structured answers were often produced that did not always accommodate more nuanced or interpretative queries.  
An area of improvement is identified in the interpretation of tabular and structured data. While the VA demonstrates adequate performance with text-based documents, limitations were exhibited in the parsing and responding to questions that required an understanding of structured data, such as schedules or course curricula. Occasional inaccuracies were observed due to the inability to seamlessly integrate tabular data, particularly in enumerative responses or when items from structured information were required to be listed by the VA. Future enhancements may be considered, including model training specifically designed for the interpretation of structured data, with the potential use of datasets containing annotated tables to enhance the model's adaptability.  
Challenges were further observed in the form of hallucinations in enumerative responses, where the model generated additional or inaccurate information when tasked with listing items or following a sequence. This tendency underscores the need for stricter filtering mechanisms to verify context relevance and accuracy, particularly in queries requiring multi-step answers. To address these hallucinations, it is suggested that enhanced prompt engineering and query design be utilized to guide the model more effectively, thereby reducing instances of fabricated information and ensuring that responses are accurate and grounded in retrieved content.  
Lastly, it has been noted that the current setup of the VA lacks a feedback mechanism for continuous learning and adaptation, which could allow for real-time adjustments based on user interactions. Incorporating a system designed to capture and process user feedback regarding response accuracy is expected to refine the model's responses over time. This would facilitate the development of an adaptive and evolving virtual assistant that is better aligned with user expectations. The model could be aided in learning from user corrections through this feedback loop, ultimately enhancing its reliability in a dynamic, real-world environment.  
Refining retrieval algorithms, incorporating feedback mechanisms, and developing specialized models for structured data processing will address the limitations. This approach will result in a more adaptable, accurate, and user-centered VA in academic settings.