# AI-Powered System to Facilitate Personalized Adaptive Learning in Digital Transformation
## 4 Results and Discussion
### Test on the Performance of LLMs with Knowledge Support  
In this section, we focus on the test to evaluate the performance of LLMs with knowledge support. To measure and compare the performance of the LLMs under different conditions, we selected questions related to different domains and compared the performance of pre-trained LLMs in answering these questions. The different questions were randomly selected from the MMLU dataset to check the response of pre-trained LLMs in diverse domains, without any extra knowledge support. We tested 4 different LLMs (Gemma2b, Gemma7b, Llama3, and Mistral) with questions from 9 different groups. For each group, questions were followed with topics from the same domain, and we calculatedthe average performance data of the LLMs. The performance of the LLMs was measured using the F1 score [65]. Figure 6 shows the conclusions of these experiments. In the figure below, the results clearly demonstrate that the different topics exerted complex and varied influences on the performance of the different LLMs. In contrast, Gemma:7b and Mistral had a comparatively more stable and better average performance on the testing sample data. More details of the experiment results can be found in Table A1 in Appendix A. To simplify the experiment and measurement, we decided to use Gemma:7b as the only default LLM in the subsequent testing experiments to test the impact of knowledge support in different domains, and this choice was based on a consideration of the representativeness and stability of the model. To gain a broader view of the impact of the knowledge domain, we tested with LLM Gemma:7b on datasets from 31 different domains. These tests aimed to examine the impact of more diverse topics on the performance of the pre-trained LLM. In the latter tests, we calculated performance metrics including accuracy, precision, recall, and F1 score. Figure 7 gives an overview of the results of these tests, and the details of the subjects tested can be found in Table A2 in Appendix A.  
Figure 6: Comparison of the performance of the LLMs based on different domain topics (X axis for groups, and Y axis for F1 score). Figure 7: The performance of Gemma:7b across different domains (X axis for groups, and Y axis for scores). Throughout all these tests, we observed that different topics imposed an strong influence on the performance of the LLMs, and the difference in model performance was mainly caused by the understanding of diverse domain-specific knowledge. Missing such related knowledge in the general model seriously limited the performance of the model in specific tasks, giving space for using expertise to improve the model. To further prove our assumption based on the impact of domain knowledge, we extend our experiments with knowledge support from the framework discussed in Section 3. Except for benchmarking the performance of the different LLMs in diverse domains, we tested the performance of the given LLM (Gemma:7b) in different domains with customized knowledge support and compared the difference with a model without extra knowledge support in the same scenarios. The purpose of these tests was to examine the potential of our proposed approach and to identify the characteristics of the framework. In these tests, we used the default LLM Gemma:7b for all agents to process prompts during the tasks. Meanwhile, the agents in the framework also used the \"paraphrase-MiniLML6-v2\" sentence transformer model [66] from the hugging-face library as the indexing model for semantic search for RAG. To test the impact of adding relevant domain-specific knowledge, we first established corresponding knowledge models in the knowledge base. These knowledge models were automatically created by the agents that extracted the knowledge from the given text-based data related to the domain-specific knowledge. In these tests, the knowledge was extracted from the files given by the users, so we tested our framework with the predefined knowledge that was prepared based on the test scenarios.