# Enhancing Online Learning Through Multi-Agent Debates for CS University Students
## 3 Materials and Methods
### Participants and Design  
In this section, we present a controlled study conducted to evaluate the potential benefits of providing students with video lectures, supplemented by video chat and LLM agents' debates. Below, we outline the study's participants, the testing conditions, and the results.  
**Participants.** We recruited 197 undergraduate and postgraduate students to participate in the study. These students, specializing in computer science, were from Hangzhou Normal University, Nanjing University of Information Science and Technology, and Hebei University of Technology. Apart from the pre-test and post-test quiz, we also provided a follow-up questionnaire. Due to the time-consuming nature of the questionnaire, a total of 90 valid responses were ultimately returned. Undergraduate students accounted for 44.4%, and postgraduate students accounted for 55.6%.  
**Study design.** We evaluated student performance on materials related to two machine learning topics. These topics were covered in two short online lecture videos by [PERSON]: one on regularization (approximately 9 min) and the other on numerical approximation of gradients (approximately 6 min). For each topic, we assessed the following three learning conditions.  
**Testing conditions.** (1) Watching only the lecture video; (2) watching the lecture video with the ability to ask questions via video-based Q&A; (3) watching the lecture video with the ability to ask questions via video-based Q&A and starting a multi-agent debate on specific topics.  
All students were randomly assigned to a different learning condition for each topic. Prior to watching the lecture videos, students completed pre-test quizzes consisting of five single-choice questions and one open-ended question per topic to assess their initial understanding of the concepts. Once students had watched the video and indicated that they were ready, we administered the post-test quiz. The post-test quiz comprised the six questions from the pre-test, along with four additional, previously unseen single-choice questions. Each correct answer earned 1 point, while incorrect answers received 0 points. All the test questions were designed by teachers who teach relevant courses. Moreover, the difficulty level was determined through tests carried out by an undergraduate and a graduate student who are beginners in this area. Most students completed the test through WeChat groups, while a smaller number participated in face-to-face testing in the laboratory. To mitigate the biases introduced by heterogeneous assessment conditions, all participants, whether recruited via WeChat groups or undergoing face-to-face testing in the laboratory, were required to complete the quiz on our runnable system ([[https://huggingface.co/spaces/dj86/VLog4](https://huggingface.co/spaces/dj86/VLog4) CustomLLMsPlusDebate]([https://huggingface.co/spaces/dj86/VLog4](https://huggingface.co/spaces/dj86/VLog4) CustomLLMsPlusDebate), accessed on 1 September 2024). Our runnable system can be considered a prototype for LLM-based online video learning. It is equipped with features such as question generation based on retrieval-augmented generation and multi-agent debate functionality.  
**Measures.** Due to differences in students' prior knowledge, we assessed the effectiveness of the learning methods by measuring the relative improvement in test scores before and after the learning videos. We calculated the relative percentage-point increase as follows:  
\[\Delta=\frac{\text{Post score }(\%)-\text{Pre score }(\%)}{\text{Pre score }(\%)} \tag{1}\]