# EduChat: An AI-Based Chatbot for University-Related Information Using a Hybrid Approach
## 6 Experimentation and Evaluation
### System Testing and Discussion  
We also conducted surveys with EduChat users using Technology Acceptance Model method [24], asking them, \"Is the system easy to use?\" with four rating levels (Very convenient, Quite convenient, Somewhat convenient, Not convenient), and \"Does the chatbot provide useful information and solve a significant portion of your problems? With four rating levels (Very useful, Quite useful, Somewhat useful, Not useful)\". These surveys were conducted from 23 September 2023, to 3 October 2023 and 28 October 2023 to 30 October 2023, for 40 first-year students at the beginning of the first semester at HUFLIT University. The evaluation level \"convenience\" ranges from fairly to very convenient, accounting for 85%, corresponding to 34 out of 40 survey participants, and the evaluation  
Figure 3: Classification results using machine learning and deep learning methods on the EduChat system.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**ID** & **Methods** & **Accuracy** \\ \hline
1 & K Nearest Neighbors (KNN) & 0.8692 \\ \hline
2 & Support vector machines (SVM) & 0.9692 \\ \hline
3 & Decision Tree (DT) & 0.9308 \\ \hline
4 & **Improved Random Forest (IRF)** & **0.9769** \\ \hline
5 & Neural Net (ANN) & 0.9538 \\ \hline
6 & Naive Bayes (NB) & 0.9538 \\ \hline
7 & LSTM [22] & 0.9688 \\ \hline
8 & BiLSTM [23] & 0.9698 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Experimental results with 08 machine learning and deep learning methods.  
level \"useful\" ranges from fairly to very useful, accounting for 90%. The results are shown in Table 4.  
The participation of the system's modules was also considered in the statistics. Out of 128 messages sent to the system, 11 were processed by the NLP module, 98 messages were classified by the machine learning module, and ChatGPT participated in processing 19 messages. Furthermore, we also conducted an analysis of 19 messages that ChatGPT processed. These were general messages about psychological issues, study methods, etc., and ChatGPT provided good advice on these issues. The integration of ChatGPT has made the system more flexible and diverse.  
From the results above, it can be seen that EduChat is a robust system based on the hybrid combination of both rule-based and machine learning methods. The rule-based approach is highly reliable, easy to verify and interpret, capable of handling limited data, and works well with incomplete data. While machine learning has the capability to learn and improve autonomously from data, achieving high performance in processing large and complex datasets, it can uncover complex patterns and adapt to diverse data without requiring expert knowledge beforehand. Furthermore, EduChat integrates ChatGPT's API, leveraging all the capabilities of ChatGPT to handle every conversation input to the system effectively.  
Currently, predictions for the future of chatbots show a strong trend of continued development. Some potential trends for the future of chatbots may include the integration of artificial intelligence (AI) and machine learning, multi-channel interaction, the development of versatile chatbots, stronger security integration, and improved user experience integration.