## 5 Evaluation and Discussion
### Sentiment Analysis of User Feedback Results  
The analysis of the sentiment results was structured to provide a comprehensive overview of the feedback sentiments, as shown in Figure 13. We began by calculating the frequency of positive, negative, and neutral sentiments for each item under review. Following this, an overall sentiment distribution analysis was carried out to understand the cumulative sentiment trends across all the received feedback. Specific to the nature of the questions, the analysis included distribution analyses for responses in the \"rate\" column, categorizing them into classifications such as \"Good\" and \"Normal\". The percentages of \"Yes/No\" responses were computed for the binary questions. In addition, for the open-ended questions, prevalent responses were identified to gauge common themes or opinions. The findings are summarized in Figures 18 and 19, showing the sentiment analysis results and overall sentiment distribution across the feedback dataset. The evaluation of 210 users' feedback following their interaction with the chatbot prototype revealed favorable outcomes. Among the 1460 sentiment expressions identified in all the questions, 1206 were positive, 184 were negative, and 70 were neutral. These findings suggest that most participants had a favorable encounter when engaging with the chatbot. The specific characteristic emphasized was the chatbot's ability to provide clear and easily comprehensible responses, which received a favorable sentiment score of 68. * Prompt and efficient response times to inquiries were recorded (69 good attitudes);
* Seamless and effortless engagement in interactive chats with the chatbot resulted in a favorable sentiment score of 70. Areas requiring enhancement encompassed the following: encountering specific challenges in engaging with the chatbot resulted in 48 instances of negative sentiment responses. Moreover, the sentiment on the extent of comprehensiveness in the FAQ subjects and alternatives varied, with only 36 instances of positive sentiment compared to 32 instances of negative sentiment. Overall, the sentiment and reaction towards the customized academic support chatbot concept and existing prototype were pretty favorable, with particular areas for enhancing the user experience. To summarize the evaluation metrics and results, conversational AI underwent an evaluation to assess its proficiency in comprehending natural language by employing a sense test. * The confusion matrix for intent classification and the distribution of prediction confidence demonstrated robust overall performance, albeit with scope for enhancement in differentiating between particular comparable intentions. * The F1 score for the majority of intentions was high, approaching 1.0, indicating a robust level of precision and recall. However, two semantically similar intentions exhibited lower F1 scores, suggesting occasional confusion errors. Additional training data augmentation can enhance the optimization of these less common intentions. * The average response time for user interactions was 967 ms, which is considered reasonably fast. However, certain specific intents, such as course equivalency, had longer response times, indicating areas that could be optimized. * Scalability testing involved up to six concurrent users and demonstrated satisfactory performance, with only slight increments in response times from 981 to 1023 ms. This suggests an ability to manage large quantities without a decrease in performance. * Positive user sentiment expressions were predominant, accounting for 83% of interactions. The main advantages were the respondents' clear and comprehensible answers as well as seamless interaction. Figure 19: Overall sentiment distribution. Figure 18: Sentiment analysis results. * Areas for improvement included difficulties faced with the chatbot and limitations in the range of FAQ topics addressed. * The positive user response underscores the need for more chatbot improvements to assist students with typical academic inquiries and procedures.