Knowledge Starts with Practice: Knowledge-Aware Exercise Generative Recommendation with Adaptive Multi-Agent Cooperation
Authors: Yangtao Zhou
Hua Chu
Yongxiang Chen
Ziwen Wang
Jiacheng Liu
Jianan Li
Yueying Feng
Xiangming Li
Zihan Han
Qingshan Li
Venue: NeurIPS 2025 poster ()
Keywords: Exercise Generative Recommendation
Knowledge State Perception
Personalized Learning
Multi-agent Cooperation
Primary area: applications

Abstract: Adaptive learning, which requires the in-depth understanding of students' learning processes and rational planning of learning resources, plays a crucial role in intelligent education. However, how to effectively model these two processes and seamlessly integrate them poses significant implementation challenges for adaptive learning. As core learning resources, exercises have the potential to diagnose students' knowledge states during the learning processes and provide personalized learning recommendations to strengthen students' knowledge, thereby serving as a bridge to boost student-oriented adaptive learning. Therefore, we introduce a novel task called Knowledge-aware Exercise Generative Recommendation (KEGR). It aims to dynamically infer students' knowledge states from their past exercise responses and customizably generate new exercises. To achieve KEGR, we propose an adaptive multi-agent cooperation framework, called ExeGen, inspired by the excellent reasoning and generative capabilities of LLM-based AI agents. Specifically, ExeGen coordinates four specialized agents for supervision, knowledge state perception, exercise generation, and quality refinement through an adaptive loop workflow pipeline. More importantly, we devise two enhancement mechanisms in ExeGen: 1) A human-simulated knowledge perception mechanism mimics students' cognitive processes and generates interpretable knowledge state descriptions via demonstration-based In-Context Learning (ICL). In this mechanism, a dual-matching strategy is further designed to retrieve highly relevant demonstrations for reliable ICL reasoning. 2) An exercise generation-adversarial mechanism collaboratively refines exercise generation leveraging a group of quality evaluation expert agents via iterative adversarial feedback. Finally, a comprehensive evaluation protocol is carefully designed to assess ExeGen. Extensive experiments on real-world educational datasets and a practical deployment in college education demonstrate the effectiveness and superiority of ExeGen. The code is available at https://github.com/dsz532/exeGen.

Knowledge Starts with Practice: Knoaledge-Aware Exercise Generative Recommendation with Adaptive Multi-Agent Cooperation
Yangtao Zhou', Hua Chu', Yongxiang Chen', Ziwen "aang"JJacheee Liu', Jianan Li';
Yueying Feng", Xiangming Li', Zihan Han', Qingshan Li'
Xidian University, 'Zhejiang University
{zhou_yt, wawa, dsz, liu_jc, xiangmli, hanzihan) estu.xidian xidian edu, cn
[ijjaanan@xxidaan edu. on
(hchu, (sshii)@mail wetamm.iaaaaaaa....aaaaaaaaaaaaa xidianx edu. cn {yueyingf) edu. cn
Abstract
Adaptive learning, which requires the in-depth understanding of students' learn- ing processes and rational planning of learning resources, plays a crucial role in intelligent education. However, how to effectively model these two processes and scamlessly integrate them poses significant implementation challenges for adaptive learning. As core learning resources, exercises have the potential to diagnose students" knowledge states during the learning processes and provide personalized learning recommendations to strengthen students' knowledge, thereby serving as a bridge to boost student-oriented adaptive learning. Therefore, introduce novel task called Knowledge-awant Exercise Generative Recommendation (KEGR). It aims dynamically infer students' knowledge states from their past exercise sponses and customizably generate new exercises. To achieve KEGR, we propose adaptive multi-agent cooperation framework, called ExeGen, inspired by the excellent reasoning and generative capabilities of L.LM-based Al agents. Specifi- cally, ExeGen coordinates four specialized agents for supervision, knowledge state perception, exercise generation, and quality refinement through an adaptive loop workflow pipeline. More importantly, we devise two enhancement mechanisms in ExeGen: I) A human-simulated knowledge perception mechanism mimics stu- dents' cognitive processes and generates interpretable knowledge state descriptions via demonstration- based In In-Context Learning (ICL). In this mechanism, a dual- matching strategy further designed to retrieve highly relevant demonstrations for reliable ICI. reasoning. 2) An exercise inteeaationnavversate laboratively refines exercise generation leveraging group of quality evaluation expert agents via iterative adversarial feedback. Finally, a comprehensive evalua- tion protocol is carefully designed to assess ExeGen. Extensive experiments on real-world educational datasets and practical deployment in college education demonstrate the effectiveness and superiority of ExeGen. The code is available at https://githhn com wttpa//coomm/me///eeeeeeee
mechanism col-
1 Introduction
Adaptive learning, which involves the in-depth understanding of students' learning processes and rational planning of learning resources, has been shown to improve learning outcomes, reduce dropout
*Correpponingg Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Knowledge
(b) Knowledge- awwaee
(c) Exercise Recommendation
Tracing
Exercise Generative Recommendation
Self Cirollire
list
Lis
F2
Knowledge
Excrelse
State
Generative
Perception (LLMs)
Recommeend
Responses
Hatorical
Exercie
KT Model
Trained
Response Prediction
Respoases
Historical
Exerrise
Interpectable
Knowledge
States
(LLMs)
-ation
Personully Generated Exercise
Hisorical Esercior Internctions
Texined ER Model
Recommended Results
Figure 1: Comparison of three intelligent education tasks.
rates, and enhance instructor satisfaction in recent empirical studies E 23 91. Consequently, it
has gained significant attention the field of intelligent education to support personalized learning. However, how to effectively model and seamlessly integrate these two stages poses significant challenges for the implementation of adaptive learning systems. As critical learning resources, exercises play pivotal role in both knowledge diagnosis and acquisition [36], which has the potential diagnose students' knowledge states (27 26] during the learning processes and provide personalized learning recommendations to strengthen students' knowledge [29] 20], thereby serving as a natural
bridge to boost student-oriented adaptive learning.
Therefore, we introduce a novel task termed Knowledge-aware Exercise Generative Recommendation (KEGR), which leverages exercises to bridge knowledge state perception and personalized learning recommendations, As illustrated in Figure [b), KEGR aims to dynamically perceive students' knowledge states from their historical exercise responses, and subsequently generate and recommend tailored exercises based on the knowledge states, thereby offering personalized learning experience. The most related tasks include Knowledge Tracing (KT) and Exercise Recommendation (ER):
KT (Figure I a)) typically predicts whether student can answer exercises correctly based on the past responses [63] [. but fails to provide interpretable knowledge state descriptions for downstream applications and often overlooks the semantics within exercises and concepts.
ER (Figure I (c) usually applies general recommendation methods to match students with exercises drawn from static, one-size-fits- exercise pool, but ignores the impact of the knowledge states, which may lead to ineffective recommendations [53 32
The inherent limitations and isolation of these two tasks hinder the realization of adaptive learning systems. KEGR addresses this by continuously perceiving students' knowledge states and customiz- ably generating new exercise recommendations to form a unified and dynamic closed-loop system, aligning with humans' long-term learning goals
Recent advances in Artificial Intelligence Generated Content (AIGC) (62]. particularly Large Lan- guage Models (LLMs) [39 622, offer a promising solution to achieve KEGR. LLMs can deeply understand semantic information and generate rich textual content [3I] allowing them to infer students' knowledge states from the answered exercises and generate personalized exercises through natural language prompts. However, due to the complexity of KEGR reasoning over student responses. exereises, and concepts, a single LLM struggles to achieve the task's adaptive goal. To address this, we propose leveraging the AI Agent technology [59 49], which excels in autonomous language interaction, role-playing, and decision-making for complex tasks. This technology can effectively stimulate LLM capabilities through multi-agent cooperation to generate high-quality knowledge-awwe awwred aware exercise recommendations.
which involves interactive
To this end, we propose Knowledge-aware Exercise Generative Recommendation Framework with Adaptive Multi-aGent Cooperation, abbreviated ExeGen. This framework integrates four specialized agents: Recommendation Manager, Knowledge Perceiver, Exercise Generator, and Quality Evaluation Expert, responsible for global supervision, knowledge state perception, exercise generation, and quality refinement, respectively. These agents collaborate through an adaptive loop workflow pipeline to generate personalized recommendations. Built on this framework, we devise two key mechanisms to enhance the precision of knowledge state perception and the quality of generated exercises. 1) human-simulattd knowledge perception mechanism (intra-agent) for the Knowledge Perceiver, mimics students' cognitive processes with domain knowledge graph to construct high-

---

quality demonstrations, enabling finer-grained tracking of their thought paths. In this way, the Knowledge Perceiver can prompt LLMs to infer the knowledge mastery of students more accurately and generate interpretable knowledge state descriptions via demonstration-base In-Context Learning (ICL.. this mechanism, a dual-matching strategy is further designed to retrieve relevant contextual demonstrations via precise and fuzzy searches, thereby boosting perception reliability. 2) An exercise generation-adversae mechanism (inter-agent) establishes a feedback-driven adversarial interaction between the Exercise Generator and Quality Evaluation Expert. Notably, group of evaluation experts, each focusing on different quality aspects, collaboratively guide the Exercise Generator to iteratively refine the generated exercises, resulting in more accurate and pedagogically personalized exercises. To evaluate ExeGen, we design a comprehensive evaluation protocol that integrates GPT- based scoring, statistical analysis, and human evaluations from both students and teachers. Extensive experiments on real-world educational datasets and a practical deployment in college education confirm the effectiveness and superiority of ExeGen, highlighting its promise for advancing adaptive learning systems. Our main contributions are summarized as follows:
We introduce novel task KEGR, and propose LLM-ppowered framework ExeGen that achieves personalized exercise generation and recommendation through adaptive multi-agent cooperation. By coordinating four specialized agents in closed-loop workflow pipeline, ExeGen bridges the gap between knowledge state perception and personalized learning recommendations.
We devise a comann-iimulate knowledge perception mechanism in Knowledge Perceiver, which enables finer-grained tracking of knowledge states through suman-simulater cognitive paths, with a dual-matching strategy for further reliable inference. This approach achieves superior transparency and interpretabillity compared to current KT methods, advancing educational diagnostic applications.
We design an exercise generation-adversaria mechanism between the Exercise Generator and Quality Evaluation Expert, where multi-dimensional experts collaboratively refine exercise generation through iterative adversarial feedback. This mechanism ensures that the generated exercises adhere rigorous educational standards while effectively addressing individual learning needs.
We design a comprehensive evaluation protocol to assess ExeGen. Extensive experiments on real- world educational datasets and an actual application demonstrate the effectiveness and superiority of ExeGen.
2 Related Work
Exercise Recommendation. Recent work adapts general recommendation technologies
22
52
64
to educational settings for personalized learning resource recommendations (58]
19
48
60
29
Effective Exercise Recommendations (ER) help guide students along appropriate learning paths and boost engagement [53). Early efforts formulate ER multi-objective optimization task. For example, Huang et al. [24] and Liu et al. (32) use deep reinforcement learning to balance review & explore, difficulty progression, and engagement. To model learning dynamics, Recurrent Neural Networks (RNNs) [53] and Long Short-Term Memory (LSTM) networks [20 21 are employed to capture students' knowledge mastery and forgetting rates. More recently, Li et al. 29 employ Graph Neural Networks (GNNs) to capture heterogeneous relations among students, exercises, and concepts, further improving the recommendation performance. Despite these advances, most existing methods overlook the nuances of individual students' knowledge states and rely only on static exercise pools, leading to one-size-fits-ae recommendations that might not align with students' personalized needs.
Knowledge Tracing. Knowledge Tracing (KT) aims to infer students' evolving knowledge states from their past exereise responses 63 Existing methods fall into two categories: traditional probabilistic methods and deep learning methods [54]. Early probabilistic methods, such as Item Response Theory (IRT) [8], Bayesian Knowledge Tracing (BKT) [8) and Factor Analysis Models (FAM) (5 7 use handcrafted features to predict student performance 2 In recent years, deep learning methods aim to learn latent patterns directly from large-scale data (3 [57]. For example, Piech et al. 42 first introduce RNNs to model response sequences, enabling dynamic tracking of knowledge states. Pandey et al.
40 incorporate self-attention to capture fine-grained knowledge states. Graph-based methods further extend KT by representing concept dependencies through GNNs
38 54 Despite these advances, current methods simplify the KT task to binary correctness prediction, which limits interpretability and reduces their potential to support personalized learning.
3

---

LLMss for Intelligent Education. Large Language Models (LLMs), such as GPTs, have significantly accelerated progress intelligent education [5] 25], supporting a wide range of applications in personalized instruction, assessment, and content generation
33 For personalized instruction, L.MSs are used to deliver Socratic-style tutoring B3 47 or to simulate interactive classroom environments through AI Agents [61]. For learning outcome assessment, LLMs help evaluate student work automatically, reducing teacher workload I0 28 For educational content generation, LLMs are used to produce textbooks, instructional images, and coding exercises aligned with curriculum goals 4 [6]. However, most of the existing exercise generation work centers on teacher-facing tools, overlooking the potential for adaptive, student-centered exercise generation 9 37.
3 Problem Formulation
Let = (ui, u2,..., uRe)) }. E = fer, E2, e2,.... f/g) and (cl, C2 ...., ,c, denote the sets of students, exercises, and knowledge concepts, respectively, For given student E U, his/her learning history is denoted as H { (eft .t . vr'), ,..,, (e) denotes the length the learning history. Here, each tuple in H consists of an excrcise e E E, its associated knowledge concept c E C, and a binary label y E (0,1} indicating whether the student answered correctly. The goal of the KEGR task is generate personalized and new exercises that align with the evolving knowledge state of each student. KEGR is achieved by addressing two tightly-coupled subtasks via natural language interactions:
c v, ,.... (e (et,e0 Gx 1/h )}. where n
Knowledge State Perception: Given a student u,'s 's exercise history H" this subtask aims generate interpretable textual description * of the student's 's current knowledge state, enabling clear understanding and actionable insights for exercise generation.
Exercise Generative Recommendation: Based on 1 this subtask aims to generate personalized exercises 204 along with their corresponding concepts C aligning well with the student's 's learning needs.
These subtasks form iterative feedback loop that continuously monitors the learning process of students and dynamically adjusts exercise generation, promoting long-term and personalized learning gains. For simplicity, omit student-specific superscript u, in the remainder of the paper.
4 Methodologies
4.1
Adaptive Multi-Agent Cooperation Framework
In this section, we present ExeGen, a framework that leverages LLMs to generate personalized knowledge-aware exercise recommendations. As shown in Figure ExeGen integrates four special- ized agents: Recommendation Manager, Knowledge Perceiver, Exercise Generator, and Quality Evaluation Expert. These agents work collaboratively in an adaptive loop workflow pipeline that follows the process of "recommendation monitoring - knowledge perception exercise generation + quality refinement". This design enables personalized and interpretable learning support through natural language interaction. The detailed prompts for all agents are listed in Appendix B.I
Recommendation Manager is the core of ExeGen, which acts as the controller and orchestrator of the entire pipeline. Inspired by recent studies (50, 35) that task decomposition strategy can significantly enhance LLMS' problem-solving capabilities, the Recommendation Manager breaks down the complex KEGR task into three sub-steps: knowledge state perception, personalized exercise generation, and exereise quality evaluation. It receives the historical exercise response data H of student uy and supervises each sub-step of the pipeline, invoking the appropriate agents in sequence. After collecting well-evaluated exercises en+1 from the final sub-step, determines whether the recommendation cycle is complete and delivers the newly generated exercises to the student u, This continuous monitoring enables an adaptive, feedback-driven learning loop.
Knowledge Perceiver focuses on prompting LLMs to analyze the historical exercise records H provided by the Recommendation Manager, thereby generating accurate and interpretable knowledge state description 8n+1 of student u;. However, LLMs often struggle with accurate knowledge inference due to hallucination issues and limited reasoning depth [4]. To mitigate these challenges, devise cumann-iimulade knowledge perception mechanism with domain knowledge graph for

---

Inpar: litssonc
II
Tusk Decompostion
Task: lisowledge- Exertiee Generative Recommendation
Infering kconlsdge
Sau stdders tased M. Gzzzratix currcoo C. lase Sast-
Students
Recommendation Manager
Aucuing fiz qualit genziaed andd parss C
Octput: Pesssalized new extrciess fail.
Cal
Inpur: ovarsic scipasse receed
Intra-Agent Mechanism
Historical Exertbe
Responses
Analyzing insoocel everise response generttigg
Knowledge
poocr inowdedie
FB
Perceiver
Output lodge desciitions Pust
Porceine
Ingurt Kcondedge ent descipiions 3as1 Tack:
Inter-Agent Mechanism
Under standing the kncwlelge atsics Sast and gaicrating tew ouacises c..t Sor the given stadents.
Evercise
Generator
Output Cuakenazed new exercess C..r
A
Iapet
escrciies Cast.
Assessing quiliy
inchrling langrage firecyy coverage mesizenneegg soncapes. Outpat: Persoraliall
Evalmation Esperts
Export Group
Workflow
Pipline
(b) Agent Role and Critical Mechanism Designs
Figure 2: Overall framework of ExeGen consists of four specialized agents.
the Knowledge Perceiver. This mechanism mimics human exercise- answering behavior that involves relevant knowledge retrieval- + analysis + exercise solving, enabling finer-grained tracking of the student's 's thought paths as well as the generation of interpretable and trustworthy descriptions of the student's knowledge states. Further details are introduced Section 4.2
Exercise Generator crafts personalized exercises tailored to the inferred knowledge states from the Knowledge Perceiver. Each exercise includes stem, associated concept, and standard answer. To ensure knowledge state alignment and reasoning quality, we integrate Chain-of-Thouge (CoT) (50) prompting during exercise generation. The Exercise Generator also supports prompt-level control through two configurable parameters: exercise type and quantity. Due to the subjectivity and evaluation challenges of subjective exereise [37], we only focus on three objective exercise types: single-choice, multi-choice, and judgment, leaving the subjective exercise as our future work. For exereise quantity, the default setting 10, but can be adjusted match the learning goals of students.
Quality Evaluation Expert ensures the pedagogical quality of generated exercises through a multi- dimensional, adversarial refinement process. Drawing inspiration from the ceerraiieeddseriminativ dynamics in Generative Adversarial Networks (GANs) [14 we establish an exercise generative- adversarial mechanism between the Exercise Generator and group of specialized Quality Evaluation Experts. Considering that exercises in educational scenarios need to satisfy multi-aspect constraints, each expert targets distinet quality aspect, collectively forming multi-dimensional constraint network. This network enforces the Exercise Generator to iteratively refine its outputs until they meet all quality thresholds. Section 4.3 provides further details of this mechanism.
4.2 Human-simulated Knowledge Perception Mechanism (Intra-Agent)
While LLMs possess extensive world knowledge and strong logical reasoning capabilities [4] they face two major challenges in knowledge state perception. First, L.LMs lack domain-specific knowledge, limiting their ability capture the hierarchical relationships among courses, concepts, and exercises, which are critical for modeling students' knowledge states [17] Second, they are prone to hallucinations that compromise inference reliability in knowledge-intensive tasks [43]. To address these challenges, we propose numan-simulle aligns LLM reasoning with the multi-step cognitive process humans follow when answering exercises. Additionally, this mechanism integrates a domain knowledge graph from real-world dataset to enhance domain-speciftt knowledge retrieval and alleviate hallucinations.. As shown in Figure 3 the mechanism consists of two modules: demonstration construction and knowledge state generation. The former models human cognitive steps and embeds structured domain knowledge into contextual demonstrations, enabling step-by-step thinking processes. The latter uses demonstration-brat based ICL
knowledge perception mechanism that
5

---

Demonstration Repository
Choice Excrcises: Which the Bollowing statements is true about Java' consiructos?
Concept
Concept c,a Posttive
Concept Sy"
Poeitive
Pocitive
Relevant Knowledge Retrieval
Retrieval- ungueeted
Concept GP² Negative
Concept c,
Concept oyt
Negative
-..
Negative
Demonstration
Demonstration
Demonstration
Knwoledge Evidence:
Dusl-Matching
(iaa Trygrrming coenis, Clan
Historical Exercise Responses of Learners
curtten Static Maniona)
Excrcise Whichh keyword used class should inheit methods and fields from another class? A derive; B. estend, implemene. D. inhers. Anserr (oorax))
specify
A. oustan, Constracton)
Analsiis of Acquired Knomledge
Chain of I Thought Reasoning
2LMG epp. GPEido Chenees
In-Context I Learning Ressinting
LEAb ra GPE-As
Explamation:
C that members class.
Knowlodge Summary Learner M.
Masterel Cuncepis Usdestarinn eacapsalaiio uoing acces ccat- ni permsiios,, Recegn min elmordepe betweer closser and dbjects, Straagglng Amas Diflewstoiig betman engle and nulloplc saheni- zece winin jas; Undersaniinn Joso selass decliraton requrenents Seggesiins for imprevenents Pracical exerciees uriting and rea- any simple lnva progriess to stidity comprsaaave of the boric like (lass stumeer and data processing with the of OCP
Thie statement deed true,
A R
Solving analysis
Answer:
(n) Demonstration Construction
(b) Knowledge State Generation
Figure 3: Human-simulated knowledge perception mechanism.
to produce intermediate reasoning steps before generating the final knowledge states. This design effectively enhances interpretability, reduces hallucinations, and improves inference accuracy.
4.2.1 Knowledge Retrieval-augmentee Demonstration Construction
Prior work [13 shows that well-designed demonstrations can significantly enhance LLM reasoning. To this end, we propose knowledge retrieval-augmennt demonstration construction module craft high-quality, task-specific demonstrations prior to LLMs for knowledge state generation. As shown in Figure 3a), this module mimics the structured cognitive process humans follow when answering exercises: 1) identifying assessed concepts and retrieving relevant prior knowledge; 2) analyzing the exercises in light of the retrieved knowledge; 3) synthesizing insights derive solution. By simulating this process, the module generates demonstrations that comprehensively cover all the required concepts.
Firstly, the module employs Aatrivval-Augeentee Generation (RAG) techniques 13] to extract Knowledge Evidences (KEs) (46) from domain-specific Knowledge Graph (KG), simulating the initial step of the human cognitive process. Each KE is structured knowledge triple:
(1) where h; and L denote the head and tail entities, and represents their relationship in the KG. The configurable parameter m controls the number of retrieved triplets, affecting the depth and breadth of relevant knowledge. These KEs encapsulate critical information, such as (address mapping, is_pre, virtual address), indicating that "address mapping" a prerequisite concept for "cirtual address".
KE; = (,,,,,,,)))))) | i = 1.2,...,m
Secondly, we apply shoo CoT prompting, which uses the phrase "Let's think step by step" in the prompt to guide the LLM in generating explanatory reasoning for each knowledge concept E C and its associated exercise. This step simulates the second step of the human cognitive process along with the retrieved KEs to reduce hallucinations, which is formally defined as:
DE J. pos LLM (c (Cpos: C., KE j 1.2. = LLM (Enegy C₁ KE,) 1,2. 1.2.....
DE x i,neg
where D and DE are the LLM-generated explanations derived from the positive and negative samples, respectively. Here, Epos and Enng represent the positive sample (exercise with correct response) and negative sample (exercise with incorrect response) for concept G, respectively. This suml-sample strategy enriches the demonstrations and avoids observational bias during reasoning.

---

Finally, we construct structured demonstrations by concatenating each knowledge concept c₁, exercise instances, student responses, KEs, and the -LM-generated explanations, which simulates the final step of the human cognitive process. These demonstrations provide contextual cues that help LLMS associate knowledge understanding with student responses. The construction process is defined as:
{
DEss = epoall c
K E.] x i,po y/ DE 2 Li,negl y/
DE
(3)
DEinng enegll c:
K E
where DE ipps and D Drs Dinng denote the positive and negative demonstrations, with y: for correct responses and 3/5 = O for incorrect ones. All demonstrations are stored in a demonstration repository and used for In-Context Learning (ICL. in the subsequent knowledge state generation stage.
4.2.2 ICL-based Knowledge State Generation
To accurately infer the knowledge states of students, we introduce an ICL-based knowledge state generation module that leverages retrieved demonstrations to provide contextual support. As illus- trated in Figure (b), given a student's s historical exercise responses H, each record (eg, Cy. My) E H paired with the top-K most relevant demonstrations from the pre-built repository. Specifically, we introduce two retrieval strategies:
Hard matching strategy selects demonstrations that exactly match the concept cj from the repository. If match is found, indicating novel emerging concept, the module switches to the soft matching strategy.
Soft matching strategy firstly encode both the input record (as Query token) (ej. Cj+ Vi) and stored demonstrations via a pre-trained language model (such as Bert [12) to yield their respective semantic representations V'ueey and upg. The strategy then selects the most on their cosine similarity [56].
semantically relevant demonstrations
The retrieved demonstrations are concatenated with each exercise record, and subsequently integrated into the system prompt of the Knowledge Perceiver, thereby prompting the LLM generate both the inferred knowledge evidence KE, and step-by-step explanation 2 from student perspective. Finally, the Knowledge Perceiver synthesizes the exercise-answering explanations of historical exercise records to infer the overall knowledge state of student u4:
(KEE)))) Ü
n
Sxt1 LLM
where Uن= (-) denotes successive concatenation operation. This process aligns LLM reasoning with human cognitive processes to yield a more accurate and interpretable knowledge state Sn+1-
4.3 Exercise Generative--Adveraarii Mechanism (Inter-Agent)
The reliability of the exercises is critical for adaptive learning systems. To ensure the reliability and pedagogical quality of LLM-generated exercises, we design an exercise generative-adveret mechanism, inspired by the adversarial idea between generators and discriminators in GANs 14, 6. This mechanism enforces a and Quality Evaluation Experts. Moreover, considering that exercises educational scenarios need to satisfy multi-dimensional requirements at the same time, we introduce a group of evaluation experts, where each expert focuses on specific quality dimension: 1) linguistic fluency, coverage of erroneous knowledge concepts, and 3) overall correctness and reasonableness. These dimensions are configurable. And assigning one expert per aspect improves the clarity of the assessment. Specifically, experts assess each generated exercise and provide structured feedback. If the exercise fails any standard, the feedback is incorporated into the prompt of the Exercise Generator, guiding the LLM to revise the exercise. This process continues iteratively until the exercises satisfy all quality standards or the iterative process reaches a maximum of 10 rounds. The final refined exercise én I is then passed to the Recommendation Manager for delivery to student uy
validation loop between the Exercise Generator

---

Experiments
5.1 Experimental Setups
Datasets and Baselines. We evaluate ExeGen on the MOOCCubeX dataset [55] with KG. and the detailed dataset statistics are provided in Appendix A.1 KEGR a new task, there are no existing methods for direct comparison. Therefore, we adapt four representative LLMM-bssd approaches as
baselines: Zero-shot Learning 34 In-Context Learning (ICL) [.] Chain-of-Thought (CoT) (50]
and ICL+CoT. The implementations and prompts for all baselines are listed in Appendix A.2 and B.2
Evaluation Metries. To comprehensively assess ExeGen, design a three-step evaluation protocol:
1) offline hybrid scoring. 2) online post-questionnaires from the student perspective, and 3) online expert evaluation from the teacher perspective. Motivated by prior research [30], which shows that GPT-based evaluation can align with manual evaluation on content generation tasks, we incorporate both GPT-based and statistical metrics in the offline scoring step. Specifically, we design six GPT- based metrics tailored to KEGR: Knowledge Relevance (KR), Clarity, Answer Accuracy (AA), Difficulty Appropriateness (DA), Engagement and Fun (EF), and Safety and Ethics (SE). Meanwhile, we design two statistical metrics: Error Hit Rate (HR) and Error Recall (Recall). GPTT-based metrics are scored on 0-5 scale, while statistical metrics range from 0 to with higher scores indicating better performance. Detailed implementations and prompts of evaluation metrics are provided in Appendix and B.3 In addition, we develop a dual-interface system based on ExeGen for both teachers and students, which has been deployed in real-world college scenarios (see Appendix C
and
5.2 Performance Comparison
To verify the superiority of ExeGen, we evaluate against four baselines on three exercise types: single choice, multiple choice, and judgment. The experimental results summarized in Table with optimal results highlighted in bold and suboptimal results marked with an underlined. Based on the results, find the following key conclusions: Firstly, ExeGen consistently outperforms all baselines across all exercise types, demonstrating its superior capability to capture student knowledge states and generate personalized exereise recommendations. These results further validate the effectiveness of integrating multiple specialized agents, which provide more reliable generation than direct prompt-basee methods. Secondly, among the baselines, Zero-shot performs the worst, while ICL and CoT show clear improvements. This result aligns with previous findings [ (50] that in-context demonstrations and stepwise reasoning help LLMs better understand tasks and follow instructions. Finally, ICL+CoTT achieves the sub-optimal performance, surpassing both the ICL. and COT. This result highlights the complementary benefits of integrating in-context learning with stepwise reasoning for improving the logical inference and task comprehension capabilities of LLMs.
Table 1: Performance comparison of different methods.
Types
Methods
KR
Clarity
AA
DA
EF
SE
HR
Recall
Zero_shot
3.77
3.67
3.69
3.44
3.28
4.27
0.11
0.20
ICL.
4.01
4.09
4.13
4.10
4.01
4.21
0.29
0.35
Single Choice Exercise
COT
4.38
3.87
3.91
3.86
3.82
0.31
0.40
ICL+COT
4.07
4.35
4.51
4.24
4.09
4.32
0.19
0.45
ExeGen
4.81
4.78
4.89
4.81
4.72
4.62
0.70
0.75
zero_shot shot
3.63
3.62
3.49
3.60
3.41
4.11
0.08
0.15
ICI.
4.07
3.99
3.99
4.09
3.82
4.25
0.19
0.30
Multiple Choice Exercise
COT
4.06
4.25
4.09
4.05
3.69
4.02
0.30
0.35
ICL+COT
4.25
4.40
4.39
4.28
4.11
4.34
0.21
0.40
ExeGen
4.82
4.73
4.79
4.80
4.58
4.69
0.79
0.82
zcro_shot
3.77
3.63
3.66
3.29
3.22
3.83
0.11
0.18
ICL
3.84
4.22
4.21
4.15
4.25
3.96
0.20
0.28
Judgment Exercise
COT
3.94
4.29
3.86
3.88
3.68
3.82
0.12
0.30
ICL+COT
4.03
4.54
4.57
4.24
4.10
4.23
0.31
0.42
ExeGen
4.71
4.61
4.62
4.80
4.66
4.63
0.65
0.72

---

5.3 Ablation Study
We conduct ablation studies to assess the impact of different key components by removing each component individually while keeping the others unchanged, resulting in five variants: w/o. KP removes the Knowledge Perceiver (KP), directly generating exercises based on students' historical responses via other agents. w/o. EG removes the Exercise Generator (EG) and prompts the Recom- mendation Manager to generate exercises directly according to the Knowledge Perceiver outputs. w/o. QEE removes the Quality Evaluation Experts (QEEs), presenting exercises to students without further assessment and refinement. w/o. HKP removes the Human--imulatte Knowledge Perception mechanism (HKP), directly generating the descriptions of knowledge states in single step based only on students' past exercise responses. w/o. EGA eliminates the Exercise Generative-AAversarial mechanism (EGA), only using single evaluation expert without multi-dimensional refinement.
Table 2: The ablation performance ExeGen on multiple choice exercises.
Methods
KR
Clarity
AA
DA
EF
SE
HR
Recall
w/o. 3.26
3.40
3.31
3.32
3.25
3.27
0..0
0.25
w/o. EG
3.70
3,92
3.73
3.77
3.98
3.24
0.29
0,30
w/o. QEE
4.31
4.21
4.27
4.14
4.31
4.13
0.30
0.35
w/o. HKP
3.53
3.46
3.50
3.58
3.50
3.54
0.20
0.26
EGA
4.37
4.42
4.37
4.38
4.46
4.37
0.41
0.40
ExeGen
4.82
4.73
4.79
4.80
4.58
4.69
0.79
0.82
Table presents the performance of each variant against ExeGen in the multiple-choice exercise scenario. The results indicate that removing any component leads to significant drop across all eight evaluation metrics, highlighting the indispensable role of each component. Among them, w/o. KP exhibits the largest performance degradation, validating the critical role of accurate knowledge state modeling in guiding effective exercise generation. w/o. QEE further emphasizes the necessity of quality control. Without rigorous assessment and refinement, the generated exercises might suffer from limited practical effectiveness. Additionally, the significant drop of w/o. HKP confirms that sim- ulating human-like perception effectively improves the L.MM's understanding for student knowledge states. Lastly, the performance drop of w/o. EGA confirms that iterative expert feedback useful for improving exercise quality. In conclusion, these findings collectively affirm that each component is essential for generating accurate, personalized, and high-quality exercise recommendations.
AA
Clarity
Clarity
WA
Clarity
4.3
42
42
DA
KR
DA
KR
DA
KR
Single Choice
Multiple Choice
Judgment
Queen
Queen
1.00
1.00
1.00
Single Choice
Multiple Cheice
0.75
0.75
Judgmeat
0.75
0.50
11.50
0.50
0.25
0.25
0.25
HR
Recall
HR
Recall
HR
Recall
Figure 4: Influences of different LLM types.
5,4 Further Analysis on LLM types
We examine the impact of different base LLMs on ExeGen's performance using LLaMa-70B, Qwen- turbo-1101 1101, and GPT-4o. As shown in Figure all base models perform well across all evaluation metries, confirming the robustness of ExeGen's knowledge state perception and iterative generation strategies. Among them, toventtubb-1100 and GPT-4o achieve the strongest and comparable

---

performance, highlighting their advanced language understanding and reasoning capabilities. In this paper, we use GPT-4o to implement ExeGen. Besides, we further explore the impact of different LLM scales and conduct two kinds of human evaluations, which are detailed in Appendix A.5
5.5 Case Study
We conduct case study using a randomly sampled student from the MOOCCubeX dataset. As shown in Figure 5 Knowledge Perceiver produces an interpretable diagnostic report detailing mastered concepts, weak areas, and suggestions for improvement. For example, although the student answers exercises on hard and symbolic links correctly in isolation, he/she struggles with exercises combining both, indicating a gap in integrated understanding, Besides, to enhance exercise quality, we explicitly design the prompt of Exercise Generator to encourage variation in wording, difficulty, and context: "Ensure diversity in wording, difficulty levels, and scenarios to maintain the student's 's engagement and provide an appropriate level of challenge". The report of Evaluation Experts confirms that the generated exercises are well-aligned with the diagnosed weaknesses and display strong diversity. For example, exercises 3, 5, and 9 undergo iterative refinement to target different aspects of the hard links and symbolic links, including correct usage, cross-partition behavior, and conceptual limitations.
Historical
Exercise
Performance
Catalogues
Carrent
Symbolic
Links
Login
Annibutes
Catalogues
Carrent
Hand Links
Hard Links
[Hladd Links
Synbolic Links
Compression
Output of Knowledge Perceiver
Which the following descriptions conrect about the relationship between the file land link and the source lile?
Exercise:
Mastered Knowledge Concepts:
The student has demonarzted stong inderstanding following """hhwnn' Command"*
Option File hand link and source file actually slare the same file content data, B. hard link; Hard links of files hard disk inother partition, Changes
student consistenlly identifis group correct systar for changing group ounerdhip of files.
partition dolation
effeet
hard contert
"*Symboli Links ('In'>n The andaii canseelly alaniiicc that sinbolic
will not offect the scurce file.
Knowledge Evidence: (K_ Hand Fink, belung to C. linx System Adhinistaaions)
links are required to cross diffzrent file systems and understands their
Explanation The students ameoerr wae int The correct options A and B. Optice correct because deleting ociginal does not affect hard since both reference
behavice when the
Areas Struggle.
file deletel
Uption
the
The studeat eslibuts paps misundrsitnduug in the following arcss HHard Links vs. Symbolic Links"*:
the same inude Option C is incorrcol because hard links camnol span actoss different juiiiions Opiion D iusrree because clunges lo one affict the oller due to slancal dats.
Despite cornectly identifying
repects of hard links (eg. shanng the
inoes), the student repentally selests incoreet options rolated
tikks spanting pariiiions affecting source
upon deletion.
Output of Evaluation Experts
Confusion exists betwees hard links and symbolic links, particularly concerning thair ahilit to span file gatems and hundle deletions.
Expert
Expert 2:: Idlentify Missing or Repeated Knowledge Concept
*AAvvanced Usage of 'cp" Suggestions for Improvement To address these gaps and eshance the student's overall understanding, the following stepo are recommenedd
"* MMssiing Concepts** Nonz of the weak knowlelge concapts identified the so be
fiom aultiple exercises target 'pl,, 'szip' and mybotic links, each exerrine intoodacss a different aepect of these topics For esample:
"Repeated Concepts*
1. Clariif Hard vs. Symbolic Linkg**.
EExrciiee 1 and Esencise 8 both focus in 'pp. bu they aálres different flags (-yl vs. 'R)).
Provide elear explanationa inoee, no pross-partttin support) from symbolic links (poincers, cross- partition cagable).
disanguishing hard links (same
Exercisee 3, 5. and 9 all involve hard sysiiii links, bui they cover distinct nspects behavicr, pariiiin capability, and limitatons).
Thus, regettioon justified ensures comprebatsive covarage manncdd concepts.
2. Reeiiw Advanced Usages of Key Commands ......
Exppert
Figure 5: ExeGen generates a knowledge state report and a exercise quality evaluation report.
6 Conclusion
This paper introduces novel task KEGR, which aims to perceive students' knowledge states and generate personalized exercise recommendations for supporting adaptive learning. To achieve KEGR, we propose ExeGen, a multi-agent cooperation framework built on LLMs, where four specialized agents collaborate in an adaptive loop workflow. Moreover, we devise two key mechanisms: a human- simulated knowledge perception mechanism to achieve interpretable inferences of knowledge states, and an exercise generative-adversnn dimensions. Finally, extensive experiments on real-world datasets and an actual application confirm the effectiveness of ExeGen. The limitations and future work are detailed in Appendices D
mechanism to refine the generated exercise from multiple
Acknowledgments
This work was supported by the National Natural Science Foundation of China (U21B2015 and 62372351), Natural Science Foundation of Jiangsu Province (BK20232028). the Young Scientists Fund of the National Natural Science Foundation of China (62202356 and 62302373). Intelli- gent Financial Software Engineering New Technology Joint Laboratory Project (9990122088) the Fundamental Research Funds for the Central Universities (QT7X24072), and the Artificial Doolligence-Empowere Course Reform Project of Xidian University (ZNB2409).
10

---

References
[1] Abdelrahman Ghodai, Wang Qing. Deep graph memory networks for forgetting- knowl- edge tracing // IEEE Transactions on Knowledge and Data Engineering. 2022. 35, 8. 7844-7855.
[2] Abdelrahman Ghodai, Wang Qing, Nunes Bernardo. Knowledge tracing: A survey // ACM Computing Surveys. 2023. 55, 11. 1-37.
[3] Bailey Allison, Vaduganathan Nithya, Henry Tyce, Laverdiere Renee. Pugliese Lou. Making digital learning work: Success strategies from six leading universities and community colleges
I/ Boston: Massachusetts: Boston Consulting Group. 2018.
[4] Caines Andrew, Benedetto Luca, Taslimipoor Shiva, Davis Christopher, Gao Yuan, Andersen Oeistein, Yuan Zheng, Ellion Mark, Moore Russell, Bryant Christopher, others On the application of Large Language Models for language teaching and assessment technology arXiv preprint arXiv:2307 .08993. 2023.
[5] Cen Hao, Koedinger Kenneth, Junker Brian. Learning factors analysis a general method for cognitive model evaluation and improvement // International conference on intelligent tutoring systems. 2006. 164-175.
[6] Chen Hao, Wong Zefan, Huang Feiran, Huang Xiao, Xu Yue, Lin Yishi, He Peng, Li Zhoujun. Generative adversarial framework for cold-start item recommendation // Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2022. 2565-2571.
[7] Instructional factors analysis: A cognitive model for multiple instructional interventions. //. 2011.
[8] Corbett Albert T, Anderson John R. Knowledge tracing: Modeling the acquisition of procedural knowledge User modeling and user-adapted interaction. 1994. 4. 253-278.
[9] Cui Peng, Sachan Mrinmaya. Adaptive and personalized exercise generation for online language learning // arXiv preprint arXiv:2306.02457 2023.
[10] Dai Wei, Lin Jionghao, Jin Hua, Li Tongguang, Tsai Yi-Shan, Gašević Dragan, Chen Guanliang. Can large language models provide feedback to students? A case study on ChatGPT // 2023 IEEE international conference on advanced learning technologies (ICALT). 2023. 323-325.
[11] Deng Yang. Ren Zifeng, Zhang An. Lei Wenqiang. Chua Tat-Seng. Towards intelligent tutoring systems in online education // arXiv preprint arXiv:231 2. 2023.
[12] Devlin Jucob, Chang Ming-Way Wei, Lee Kenton, Toutanova Kristina. Bert: Pre-training of deep bidirectional transformers for language understanding // Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers). 2019. 4171-4188
[13] Dong Qingxiu, Li Lei, Dai Damai, Zheng Ce, Ma Jingyuan, Rui, Xia Heming. Xu Jingjing, Wu Zhiyong, Liu Tianyu, others A survey on in-context learning I/ arXiv preprint arXiv:2301 202424300..0024. 2022.
[14] Du Yingpeng, Luo Di, Yan Rui, Wang Xiaopei, Liu Hongzhi, Zhu Hengshu, Song Yang, Zhang Jie. Enhancing job recommendation through llm-based generative adversarial networks I/ Proceedings of the AAAI Conference on Artificial Intelligence. 38, 8. 2024. 8363-8371.
[15] Fan Wenqi. Ding Yujuan. Ning Liangbo. Wang Shijie, Li Hengyun, Yin Dawei, Chua Tat-Seng, Li Qing. A survey on rag meeting 1lms: Towards sarieval-augmennt large language models Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining. 2024. 6491-6501.
[16] Gan Wensheng, Qi Zhenlian, Wi Jiayang, Lin Jerry Chun-Wei Large language models in education: Vision and opportunities // 2023 IEEE international conference on big data (BigData). 2023. 4776-4785

---

[17] Gao Weibo, Liu Qi. Yue Linan, Yao Fangzhou, Lv Rui, Zhang Zheng. Wang Hao, Huang Zhenya. Agent4Edu: Generating Learner Response Data by Generative Agents for Intelligent Education Systems // arXiv preprint arXiv:2501. 100.:22311.032... 2025.
[18] Georg Rasch. Probabilistic models for some intelligence and attainment tests // Copenhagen: Institute of Education Research. 1960.
[19] Gong Jibing, Wang Shen, Wang Jinlong, Feng Wenzheng, Peng Hao, Tang Jie, Yu Philip Attentional graph convolutional networks for knowledge concept recommendation in moocs heterogeneous view // Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval. 2020. 79-88.
[20] Guan Quanlong, Cheng Xinghe, Xiao Fang, Li Zhuchou, He Chaobo, Fang Liangda, Chen Guanliang, Gong Zhiguo, Luo Weiqi. Explainable exercise recommendation with knowledge graph // Neural Networks. 2025. 183. 106954.
[21] Guan Quanlong, Xiao Fang. Cheng Xinghe, Fang Liangda, Chen Ziliang, Chen Guanliang, Luo Weiqi. Kg4ex: An explainable knowledge graph-based approach for exercise recommendation // Proceedings of the 32nd ACM international conference information and knowledge management. 2023. 597-607.
[22] He Xiangnan, Deng Kuan, Wang Xiang, Li Yan. Zhang Yongdong, Wang Meng. Lightenn: Simplifying and powering graph convolution network for recommendation // Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. 2020. 648.
[23] Recommendation acceptance in simple adaptive learning system. 2019.
[24] Huang Zhenya, Liu Qi, Zhai Chengxiang, Yin Yu, Chen Enhong. Gao Weibo, Hu Guoping. Exploring multi-objective exercise recommendations in online education systems // Proceedings of the 28th ACM international conference information and knowledge management, 2019. 1261-1270.
[25] Hwang Sunhee, Kim Yudoo, Lee Heejin. ChatGPT and Its Educational Impact: Insights from a Software Development Competition // arXiv preprint arXiv:2409.03779, 2024.
[26) Lee Unggi, Bae Jiyeong, Kim Dohee, Lee Sookbun, Park Jaekwon, Ahn Taekyung, Lee Gunho, Stratton Damji, Kim Hyeoncheol. Language model can do knowledge tracing: Simple but effective method to integrate language model and knowledge tracing task // arXiv preprint arXiv:2406.0228 028933 2024.
[27] Li Jiawei, Deng Yuanfei, Qin Yixiu, Mao Shun, Jiang Yuncheng. Dual-Channel Adaptive Scale Hypergraph Encoders With Cross- View Contrastive Learning for Knowledge Tracing IEEE Transactions on Neural Networks and Learning Systems. 2024.
[28] Li Qingyao, Fu Lingyue, Zhang Weiming, Chen Xianyu, Yu Jingwei, Xia Wei, Zhang Weinan, Tang Ruiming, Yu Yong. Adapting large language models for education; Foundational capabili- ties, potentials, and challenges // arXiv preprint MxXiv:2401.08666 2023.
[29] Li Shufei, Liu Xingwu, Tang Xiaolan, Chen Xi, Pu Juhua. MLKT4Rec: Enhancing Exercise Recommendation Through Multitask Learning With Knowledge Tracing IEEE Transactions on Computational Social Systems. 2024.
[30] Lin Bill Yuchen, Ravichander Abhilasha, Lu Ximing, Dziri Nouha, Sclar Melanie, Chandu Khyathi, Bhagavatula Chandra, Choi Yejin. The unlocking spell on base Ilms: Rethinking alignment via in in-conextt learning // arXiv preprint arXiv:2312.0155: 2023.
[31] Lin Jianghao, Dai Xinyi, Xi Yunjia, Liu Weiwen, Chen Bo, Zhang Hao, Liu Yong, Wu Chuhan Xiangyang, Zhu Chenxu, others How can recommender systems benefit from large language models: A survey // ACM Transactions on Information Systems. 2025. 43, 2. 1-47.
[32] Liu Fei, Hu Xuegang, Liu Shuochen, Bu Chenyang, Wu Le. Meta multi-agent exercise rec- ommendation: A game application perspective // Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023, 1441-1452.
12

---

[33] Liu Jiayu, Huang Zhenya, Xiao Tong, Sha Jing. Wu Jinze, Liu Qi Wang Shijin, Chen Enhong. SocraticLM: exploring socratic personalized teaching with large language models // Advances in Neural Information Processing Systems. 2024. 37. 85693-85721.
[34] Liu Junling, Liu Chao, Zhou Peilin, Lv Renjie, Zhou Kang, Zhang Yan. Is chatgpt a good recommender? a preliminary study arXiv preprint arXiv:2304. 10149. 2023.
[35] Liu Qidong, Wu Xian, Wang Yejing, Zhang Zijian, Tian Feng, Zheng Yefeng, Zhao Xiangyu. Llm-esr: Large language models enhancement for long-tain tailed sequential recommendation // Advances in Neural Information Processing Systems. 2024. 37. 26701-26727.
[36] Liu Zitao, Guo Teng, Liang Qianru, Hou Mingliang, Zhan Bojun, Tang Jiliang, Luo Weiqi, Weng Jian. Deep Learning Based Knowledge Tracing: A Review, A Tool and Empirical Studies // IEEE Transactions on Knowledge and Data Engineering. 2025.
[37] Logacheva Evanfiya, Hellas Arto, Prather James, Sarsa Sami, Leinonen Juho. Evaluating contextually personalized programming exercises created with generative AI Proceedings of the 2024 ACM Conference on International Computing Education Research- Voluuee 1. 2024. 95-113.
[38] Nakagawa Hiromi, Iwasawa Yusuke, Matsuo Yutaka. Graph-based knowledge tracing: modeling student proficiency using graph neural network // IEBE/WIC//CM international conference on web intelligence. 2019. 156-163.
[39] Ouyang Long, Wu Jeffrey, Jiang Xu, Almeida Diogo Wainwright Carroll, Mishkin Pumela, Zhang Chong, Agarwal Sandhini, Slama Katarina, Ray Alex, others. Training language models to follow instructions with human feedback // Advances in neural information processing systems. 2022. 35. 27730-27744
[40] Pandey Shalini, Karypis George. A self-attentive model for knowledge tracing // arXiv preprint 11Xiv:1907006837 2019.
[41] Pavlik Philip I, Cen Hao, Koedinger Kenneth R. Performance factors analysis- a new alternative knowledge tracing // Artificial intelligence in education. 2009. 531-538.
[42] Piech Chris, Bassen Jonathan, Huang Jonathan, Ganguli Surya, Sahami Mehran, Guibas Leonidas J. Sohl-Dicksteni Jascha. Deep knowledge tracing // Advances in neural information processing systems. 2015. 28.
[43] Ren Xubin, Tang Jiabin, Dawei, Chawla Nitesh, Huang Chao. survey of large language models for graphs // Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024. 6616-6626.
[44] Ren Xubin, Wei Wei, Xia Lianghao, Su Lixin, Cheng Suqi, Wang Junfeng, Yin Duwei, Huang Chao. Representation learning with large language models for recommendation // Proceedings of the ACM Web Conference 2024. 2024. 3464-3475.
[45] Tu Tao, Schaekermann Mike, Palepu Anil, Saab Khaled, Freyberg Jan, Tanno Ryutaro, Wang Amy, Li Brenna, Amin Mohamed, Cheng Yong. others Towards conversational diagnostic artificial intelligence // Nature. 2025. 1-9.
[46] Wang Jianing, Sun Qiushi, Li Xiang, Gao Ming. Boosting language models reasoning with chain-of-knowledge prompting arXiv preprint arXiv:2306.06427, 2023.
[47] Wang Tianfu, Zhan Yi, Lian Jianxun, Hu Zhengyu, Yuan Nicholas Jing. Zhang Qi. Xie Xing, Xiong Hui. LLM-powered Multi-agent Framework for Goal-oriented Learning in Intelligent Tutoring System // arXiv preprint arXiv:2501.15749, 2025.
[48] Wang Xinhua, Ma Wenyun, Guo Lei, Jiang Haoran, Liu Fangai, Xu Changdi HGNN: Hyperedge-hee graph neural network for MOOC course recommendation // Information Processing Management. 2022. 59, 3. 102938.
[49] Wang Zhefan, Yu Yuanging, Zheng Wendi, Ma Weizhi, Zhang Min. Multi-Agent collaboration framework for recommender systems // arXiv e-prints. 2024. arXiv-2402.
13

---

[50] Wei Jason, Wang Xuezhi, Schuurmans Dale. Bosma Maarten, Xia Fei, Chi Ed, Le Quoc V. Zhou Denny, others. Chain-of Ohain-of-thought prompting elicits reasoning in large language models // Advances in neural information processing systems. 2022. 35.
[51] Wen Qingsong, Liang Jing, Sierra Carles, Luckin Rose, Tong Richard, Liu Zitao, Cui Peng, Tang Jiliang. Al for education (AI4EDU): Advancing personalized education with LLM and adaptive learning // Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024. 6744.
[52] Wu Jiancan, Wang Xiang, Feng Fuli, He Xiangnan, Chen Liang, Lian Jianxun, Xie Xing, Self- supervised graph learning for recommendation // Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval. 2021. 726-735.
[53] Wu Zhengyang, Li Ming, Tang Yong, Liang Qingyu. Exercise recommendation based on knowledge concept prediction // Knowledge-Baad Based Systems. 2020. 210. 106481.
[54) Xu Bihan, Huang Zhenya, Liu Jiay,, Shen Shuanghong, Liu Qi Chen Enhong, Wu Jinze, Wang Shijin. Learning behavior-oriented knowledge tracing // Proceedings of the 29th ACM SIGKDD conference on knowledge discovery and data mining. 2023. 2789-2800.
[55] Yu Jifan, Wang Yuquan, Zhong Qingyang, Luo Gan, Mao Yiming, Sun Kai, Feng Wenzheng, Xu Wei, Cao Shulin, Zeng Kaisheng, others MOOCCubeX: large knowledge-centered repository for adaptive learning in MOOCs // Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 2021. 4643 46522.
[56] Zhang Baoquan, Li Xutao, Ye Yunming, Huang Zhichao, Zhang Lisai. Prototype completion with primitive knowledge for few-shot learning // Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021. 3754-3762.
[57] Zhang Jiani, Shi Xingjian, King Irwin, Yeung Yan. Dynamic key-value memory networks for knowledge tracing // Proceedings of the 26th international conference on World Wide Web. 2017. 765-774.
[58] Zhang Jing, Hao Bowen, Chen Bo, Li Cuiping, Chen Hong, Sun Jimeng. Hierarchical reinforce- ment learning for course recommendation in MOOCs // Proceedings of the AAAI conference on artificial intelligence. 33, 01. 2019, 435-442.
[59] Zhang Junjie, Hou Yupeng, Xie Ruobing, Sun Wengi, McAuley Julian, Zhao Wayne Xin, Lin Leyu, Wen Ji-Rong. Agentef: Collaborative learning with autonomous language agents for recommender systems // Proceedings of the ACM Web Conference 2024. 2024. 3679-3689
[60] Zhang Si, Hui Ning, Zhai Peiyun, Xu Jiali, Cao Lanying, Wang Qiyun. A fine ane-grained and multi-context-awat learning path recommendation model over knowledge graphs for online learning communities // Information Processing & Management, 2023. 60, 5. 103464.
[61] Zhang Zheyuan, Zhang-Li Daniel, Yu Jifan, Gong Linlu, Zhou Jinchang, Hao Zhanxin, Jiang Jianxiao, Cao Jie, Liu Huiqin, Liu Zhiyuan, others Simulating classroom education with llm-eepweered agents // arXiv preprint 1xXiv22406.19226. 2024.
[62] Zhao Wayne Xin, Zhou Kun, Li Junyi, Tang Tianyi, Wang Xiaolei, Hou Yupeng, Min Yingqian, Zhang Beichen, Zhang Junjie, Dong Zican, others A survey of large language models // arXiv preprint arXiv:2303. 18223. 2023. 1.2.
[63] Zhao Weizhong, Xia Jun, Jiang Xingpeng, He Tingting. A novel framework for deep knowledge tracing via gating-com controlldd forgetting and learning mechanisms // Information Processing & Management. 2023. 60, 1. 1031 14.
[64] 7hu Yunqin Wang Chao, Zhang Qi, Xiong Hui. Graph signal diffusion model for collaborative filtering // Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024. 1380-1390.
14

---

Appendices
Experiment Details
15
A.1 Dataset Statistics
15
A.2 Baseline Models
15
A.3
Three-step Evaluation Protocol
16
A.4
Implementation Details and Financial Costs
17
A.5
Supplementary Experiments
17
A.5.1
Further Analysis LLM scales
17
A.5.2
Online Post-Questionnaire (Student Perspective)
18
A.5.3
Online Expert Evaluation (Teacher Perspective)
18
B
Prompt Designs
18
B.1
Agent Prompt Templates
18
B.2
The prompt of Baseline Models
22
B.3
GPT-based Scoring Prompt
25
Real-world Deployment
26
D Limitations and Future Work
26
A Experiment Details
A.1 Dataset Statistics
To evaluate the effectiveness of ExeGen, we use the public MOOCCubeX dataset [55], sourced from one of China's 's largest MOOC platforms. Given its rich content and extensive user base, MOOCCubeX widely used in educational data mining, particularly for online learning research. The statistical details are provided in Table 3 Following prior studies [58] 19 48), we extract all student interaction data from the computer science and technology domain for our experiments.
Table Statistics the collected dataset.
Entities
#Entities
Relations
#Relations
Course
63
Course- Concept
52451
Student
18638
Pvercise-Concepp
4204
Knowledge Concept
27918
Student-Course
101007
Exercise
5015
Student-Fxeree
6153
A.2 Baseline Models
This section details the four baseline models adapted for comparison with our ExeGen framework for the Knowledge-aware Exercise Generative Recommendation (KEGR) task, all of which are implemented using the GPT-40 API interface without involving any large-model training fine- tuning. The prompts for these basclines are provided in Appendix B.2
uses task-specific prompts to directly generate exercise recommendations based on the LLMs* world knowledge and natural language understanding capabilities.
Zero-shot Learning (deroshhot): This baseline 33
15

---

In-Context Learning (ICL): This baseline [1]
provides LLMs with task-relevant contex- tual information, such as demonstrations and structured knowledge descriptions, enabling them to infer underlying patterns and generate exercises accordingly.
Chain-of-Thoughh (CoT): This baseline [30] decomposes the exercise generation process into intermediate inference steps, simulating the human thought process enhance logical coherence and interpretability.
This hybrid baseline combines ICL and CoT techniques. It first enriches LLMs with contextual information and then guides them through stepwise reasoning process to generate exercise recommendations.
ICL CoT:
A.3 Three-step Evaluation Protocol
To comprehensively assess the proposed ExeGen, we design a three-step evaluation protocol, includ- ing offline hybrid scoring, online post-questionnaires from the student perspective, and online expert evaluation from the teacher perspective.
Step 1: Offline Hybrid Scoring. Extensive empirical studies [30] have demonstrated strong correlation and consistency between GPT-based scoring methods and human evaluation, making it a viable alternative for large-scale task performance assessment without requiring manual annotation. Moreover, GPT-based scoring methods exhibit adaptability to various natural language generation evaluation tasks through the design of task-specific prompts. Therefore, systematically assess the effectiveness our proposed ExeGen for the KEGR task, we introduce a hybrid scoring approach that combines GPT-based scoring (using advanced GPT-4o) with statistical methods, regarding the following eight aspects:
Knowledge Relevance (KR): A GPT- metric that measures the alignment between the generated exercise content and the student's current knowledge states.
Clarity: A GPT-based metric assessing whether the exercise stem and answer are clearly formulated and unambiguous.
Answer Accuracy (AA): A GPT-based metric evaluating whether the generated exercise answer is unique and correct.
Difficulty Appropriateness (DA): A GPT-based metric assessing whether the difficulty level of the exercises well-calibrated to the students' proficiency, providing fitting challenge.
Engagement and Fun (EF): A GPT-based metric that assesses whether the format, content, and contextual designs of the exercises foster student interest and active participation.
Safety and Ethics (SE): A GPT-based metric ensuring that the exercise content does not contain potentially harmful information or violate ethical and moral standards.
Error Hit Rate (HR): A statistical metric quantifies the fraction of non- mastered knowledge concepts among all concepts tested by the newly generated exercises:
Error Recall (Recall): statistical metric measures the proportion of unique non-mastered knowledge concepts that are contained in the generated exercises out of all non-mastered concepts the student's history:
GPT-based metrics are scored on a 0-5 scale, while statistical metrics range from 0 to 1, with higher scores indicating better performance. The prompts of these GPT-based metrics are provided in Appendix B.3. The HR is defined as:
He | E Cnew, value(c.) = 0}l
HR
(5)
[[ c Cnewl
where Crew the set of all knowledge concepts covered by the generated exercises and value(c;) O indicates that is non-mastered (i.e., error-answered). The Recall is defined as:
Hcs c; E Crees volue(c;) (c;) = 0}
Recall
(6)
[G 04 E Chestr value (c:) 0)
where Chast denotes the set of all knowledge concepts the student's historical record and Crea denotes the subset of those concepts that are contained in the generated exereises.
16

---

Step 2: Online Post- Post-Questions sust-Questionnirres from Student Perspective. To further assess ExeGen, we recruit 20 college student volunteers to interact with the ExeGen-based system and complete post-questionnaire. The questionnaire covers five key aspects: knowledge relevance, clarity, answer accuracy, difficulty appropriateness, and engagement and fun. This step evaluates the recommendation effectiveness and user experience of ExeGie in supporting personalized learning from the student perspective.
Step 3: Online Expert Evaluation from Teacher Perspective. In this step, we invite three college teachers the field of computer science and technology as experts to manually evaluate the quality of exercise generative recommendations for 20 randomly sampled student groups in the MOOCCubeX dataset. Using the same five evaluation aspects as in step this step provides a more authoritative assessment on the pedagogical quality and practical value of ExeGen from the teacher perspective.
A.4 Implementation Details and Financial Costs
We implemented our proposed ExeGen using non-distributed training Python 3.8.19 and PyTorch 2.3.0. All experiments were conducted on Linux machine configured with two 4090 GPUs. We selected GPT-4o as our base LLM and used the OpenAl API, without fine-tuning applied. In our experiments, generating knowledge-basee exercise recommendations for each student using the GPT-4o API incurs an average financial cost of 0. 1787 dollars.
A.5
Supplementary Experiments
A.5.1 Further Analysis on LLM scales
To analyze the impact of LLM scale on ExeGen's performance, compare three variants using different-scale Qwen model: Qwen-turbo (lightweight), Qwen-plus (medium-scalbl and Qwen-max (largest-scale). As shown in Figure 6 performance improves with model scale, but the gain from Qwen-plus to Qwen-max smaller than that from Qwen-turbo to Qwen-plus. These results align with the diminishing marginal gains of LLM scaling. This suggests strategic deployment approach: large-scale models suit high-precision tasks (e.g., exam simulation) despite higher costs, while smaller models offer balance of efficiency and performance for routine learning. These findings highlight ExeGen's ' adaptability to various educational needs and resource constraints.
5.1
5.1
5.1
KR Clirity
AA
LF
KB Clarlty
AA
EV
KR
AA
5.0
5.0
5.0
4.9
4.8
4.8
4.8
4.7
4.7
4.7
4.6
4.5
4.5
4.5
4.4
4.4
4.4
turbo plus
turbo plus
turbo plus Judgment
Single Choice
Multiple Choice
0.84
0.85
Reall
0.84
0.85
0.84
0.81
0.81
0.81
0.81
0.81
0.81
0.77
0.77
0.77
0.73
0.78
0.73
0.78
0.73
0.78
0.69
0.69
0.69
0.75
0.75
0.75
0.65
0.65
0.72
0.61
0.72
0.61
0.72
turbo plus Single Choice
turbo
turbo plus
Multiple Choice
Judgment
Figure 6: Influences of different LLM scales.
17

---

A.5.2 Online rust-Questionnaires (Student Perspective)
To further evaluate ExeGen, we recruit 20 student volunteers, each assigned a test case from the MOOCCubeX dataset matched by academic background and learning history. Following the role- actor evaluation approach from prior work [45]. each volunteer assesses his/her assigned case from student perspective. As shown Figure 7 a), ExeGen achieves high scores in knowledge relevance, clarity, answer accuracy, difficulty appropriateness, and engagement and fun, indicating the strong ability of ExeGen to generate personalized and satisfactory exercise recommendations Besides, evaluation scores of student volunteers align closely with GPT-based scores, reinforcing the reliability of GPT as a scalable assessment tool for content generation tasks.
Student scoring
Expert scoring
GPT scoring
Score
3-
3-
KR
Clarity
AA
DA
EF
KR
Clarity
DA
(a) Student Perspective
(b) Expert Perspective
Figure 7: Online manual evaluation experiments from student and teacher perspectives.
A.5.3 Online Expert Evaluation (Teacher Perspective)
We invite three college teachers in the field of computer science and technology to manually evaluate 20 randomly selected student cases from the MOOCCubeX dataset. This expert assessment offers teacher-centered perspective ExeGen's 's effectiveness. As shown in Figure (bb). ExeGen achieves high scores across all key dimensions: knowledge relevance, clarity, answer accuracy, difficulty appropriateness, and engagement and fun. These results further confirm the effectiveness and reliability ExeGen. Besides, the expert evaluation scores also closely align with the scores of GPT-based evaluation, further demonstrating the high credibility of conducting GPT-based evaluation on large-scale data.
B Prompt Designs
B.1 Agent Prompt Templates
This appendix provides the detailed prompt templates for the four specialized agents (Recommen- dation Manager, Knowledge Perceiver, Exercise Generator, and Quality Evaluation Expert) within the ExeGen framework. As shown in Table these prompt templates are designed to guide each agent executing specialized tasks, thereby enabling personalized knowledge-awan exercise recommendation pipeline.
Table 4: Prompt templates of different agents.
Recommendation Manager
You the moderator of this workflow, responsible for oversceing the collaborative process between multiple agents to create and evaluate high-quality exercises tailored to student's learning needs. After each message, please indicate which agent should speak next. Your responsibilities include the following:
1. Track Knowledge State:
18

---

Instruct Knowledge Perceiver to generate a comprchensive summary of the student's knowl- edge state, including both mastered concepts and weak aspects. Specifically, Knowledge Perceiver should:
-Analyze each exercise in the student's record, deducing reasoning for correct answers and identifying misunderstandinns for incorrect answers.
-Complete any missing or incomplete explanations, providing a breakdown of the student's thought process.
EEnure the output follows the exact format provided below, where each exercise record includes:
* *content** The content of the exercise (e.g., question text).
* **option** The options provided for the exercise (if applicable).
* **right_answer"" #: A list of the correct answers to the exercise.
** ""kkowledge_evidereeee # Multiple knowledge triples, which should represent rela-
tionships such as "The course covers knowledge concepts", "Knowledge concepts mapped to exercises", and "Knowledge concept 2 precedes knowledge concept 1".
**is_ *; A boolean indicating whether the student answered correctly. **explaaation**: A detailed explanation of why the student's answer was correct or incorrect, including the reasoning behind their answer.
-Ensure that all **nowledge _etigece*eddde*** entries are clearly formatted and correspond to the relevant knowledge concepts in the exercise, as exemplified in the provided template. -Return the summary of student's knowledge states the exact format, ensuring consistency with the example, so that it is actionable and precise for future steps.
-Important: Do not provide redundant or unnecessary information in your responses. Directly analyze and return the output as per the required format without elaborating excessively on the process.
2. Generate New Exercises: -Provide the knowledge state from Knowledge Perceiver to the exercise generation expert (Exereise Generator).
-Instruct Exercise Generator to create ten new exercises, ensuring these exercises are specifically designed around the student's weak knowledge concepts and adhere to the specified exercise type format.
-Important: Ensure that the instructions to Exercise Generator are clear and to the
point. Avoid excessive introductory or redundant statements.
3. Evaluate the Linguistic Fluency:
- -Submit the newly generated exercises Linguistic Fluency Discriminator: Verifies
whether the exercises are linguistically accurate, fluent, and clear.
4. Evaluate the Knowledge Concept Coverage:
Suubmit the newly generated exercises to Knowledge Concept Coverage discriminator:
Ensures that the exercises adequately cover the student's 's weak knowledge concepts.
5. Evaluate the Correctness and Reasonableness:
Submit the newly generated exercises to Correctness and Reasonableness discrimi-
nator: Confirms whether the exercises and their answers are accurate, logical, and suitable for the student's is current learning level.
6. Iterative Regeneration:
If any Quality Evaluation Expert finds the exercises unsatisfactory:
- Reeturn to Exercise Generator and instruct them to regenerate the exercises based on the feedback provided.
Repeat this iterative process until all three agents agree that the exercises meet the required standards.
7. Final Output:
Once all agents have approved the exercises, you need to edit the final version of the exercise list strictly in the format **o fm and return it, and say 'stopChat' to let the chat end.
8. Key Guidelines:
- Prioritize the student's weak knowledge concepts throughout the process to ensure
targeted learning.
Ensure that all steps are completed efficiently and logically, with clear communication between agents.
19

---

Manage the iterative refinement process to guarantee that the final exercises are of high quality and effectively address the student's learning needs.
Important: Ensure that all agents focus on their specific task without unnecessary repetition. Any redundant remarks should be minimized to avoid clutter and ensure an efficient workflow.
Your ultimate goal is to manage collaboration between agents and produce a final list ex- ercises that are accurate, relevant, and highly tailored to the student's 's learning requirements.
Knowledge Perceiver
You knowledge tracking expert. You will receive sample exercises and record of the student's 's performance Each record includes the student's 's response, whether it was correct or incorrect, and the associated knowledge concept. The explanation attribute represents the reasoning behind why the student answered correctly or incorrectly, but some explanations may be missing or incomplete.
Your task is as follows:
1. Analyze each exercise in the student's record:
For correct answers, deduce the reasoning or knowledge that enabled the student to answer correctly.
- For incorrect answers, identify potential misunderstandinns, gaps in knowledge, or reasoning errors that led to the mistake.
2. Complete the missing
incomplete explanations for each exercise: Clearly explain the reasoning behind the student's response or identify any misun- derstandings that led to errors.
Break down your explanation into logical steps to accurately reflect the student's thought process and understanding of the knowledge concept.
3. Aggregate Explanations:
-Generate single, cohesive summary of the student's 's overall knowledge state based
on the inferred knowledge evidences and explanations from every exercise..
4. Summarize the student's overall knowledge state of the knowledge concepts:
Identify the knowledge concepts the student has mastered based on consistent correct responses and sound reasoning.
Highlight knowledge concepts where the student struggles based on patterns of incorrect answers unclear reasoning.
SSuggest aspects for further improvement, including specific prerequisite knowledge concepts the student should review.
5. Output format:
Your output should strictly follow the format provided by the Recommendation
Manager.
Each exercise record should include the following attributes:
**content**: The content of the exercise (e.g.. question text).
** **option**: The options provided for the exercise (if applicable).
* **right_answer*t: #n The correct answer(s) to the exercise (e.g.. list of correct answers).
**knowledge_evideecc**** *; Multiple knowledge triples (e.g., "The course covers knowledge concepts", "Knowledge concepts mapped exercises", "Knowledge concept 2 precedes knowledge concept 1 "). This should represent the relationship between the topic, the exercise, and the relevant knowledge concepts.
* **is *o A boolean indicating whether the student's answer was correct or
not.
** **expanntioon *: A detailed breakdown of the student's reasoning, an explanation of why the answer was correct/incorrect.
- The summary of student's knowledge states based on all the exercise records.
20

---

Please ensure that your explanations are precise, clear, and grounded in logical reasoning to provide actionable insights into the student's knowledge state. The format should be consistent with the example provided and focus on delivering detailed yet structured response Ensure that the ** *=koowlede_evvideeeeee triples for each exercise.
includes the necessary knowledge
Exercise Generator
You are exercise generation expert. You will receive list from the Knowledge Per- ceiver that summarizes the student's knowledge state, illustrated with exercise-generation examples tagged by concept_id.
Based on this information, you will need to generate (exercise [exercise_number), new (exer- cise_ cise_type) exercises, and their answers in the format provided, ensuring that the knowledge concepts in the exercises directly correspond the provided concept id values. The exercises must strictly adhere to the specified {exercise _tpp)) format, outlined below: **exercise **
Focus generating exercises related to the student's weak knowledge concepts:
Prioritize designing exercises targeting the student's weak aspects to strengthen their understanding and improve performance.
- -Create multiple exercises related to these weak knowledge concepts to reinforce the
student's practice of these concepts.
Ensure the knowledge concepts the exercises meet the following criteria:
Each knowledge concept in the generated exercises must directly match a concept id from the student's 's historical records.
Do not create new knowledge concept names concept_io ids.
As you focus on weak knowledge concepts, ensure the generated exercises possess the following characteristics:
*Clarity** Use precise language to avoid ambiguity.
* Relevance**; Directly test the knowledge concepts mentioned summary.
** Logiccalit For choice-based exercises, ensure that distractors (incorrect options)
are relevant and reasonable, reducing the likelihood of random guessing. The generation process is as follows:
1. Analyze the summary of knowledge states (including the concept id values) to
identify the student's 's weak knowledge concepts. Then make those concepts the primary focus of your exercise design.
2. Allocate most of the exercises to the weak knowledge concepts while including
few exercises to reinforce the mastered concepts.
3. Ensure diversity in wording, difficulty levels, and scenarios maintain the student's 's
engagement and provide an appropriate level of challenge.
The generated exercises must strietly follow the (exercise_tyree __tpee format, and all knowledge concepts must directly match those provided with conceptjid, id. Your output should reflect a deep analysis of the student's learning needs and a targeted design approach.
Quality Evaluation Expert
(Using the Linguistic Fluency Discriminator as an example)
You are an exercise evaluation expert specializing in assessing linguistic fluency. You will receive list of newly generated exercises and their answers created by the exercise generation expert. Your task is to determine whether the language used in these exercises is fluent and appropriate for effective communication.
Evaluation Process:
1. ** **Eallatee Sentence Structure**
Analyze the grammatical structure of each exercise to ensure it adheres to standard language conventions.
Chheck for any grammatical errors, awkward phrasing, or incomplete sentences.
** Assess Word Choice**;
Ensure the vocabulary used is suitable for the target audience.
Identify and flag any ambiguous, overly complex, or contextually inappropriate
words.
21

---

3. *Check Coherence and Clarity**:
Confirm that the language in the exercises clearly conveys the intended meaning.
Ensure the exercises and answers are logically structured and easy to understand.
**Prvvdde Suggestions for improvement*::
Highlight specific aspects where linguistic fluency can be enhanced.
Offer recommendations for rephrasing or simplifying content without altering its
meaning.
Your evaluation should focus on ensuring that the exercises are free of language errors and effectively communicate the intended concepts.
B.2 The prompt of Baseline Models
This section presents the complete prompt templates used to implement each baseline model in experiments. For each model, we provide the exact prompts used, including instructions, input formats, and any demonstrations or examples provided to the LLMs, as shown in Table These templates are carefully designed to maximize each baseline's performance for fair comparison with our proposed framework, ensuring that differences in results reflect genuine methodological advantages rather than prompt engineering artifacts.
Table 5: Prompt templates of different baselines.
Zero-shot Learning (Zero_shop shoo)
You are an expert in generating high-quality single-choice exercises for educational pur- poses. Your task is to create 10 single-choice exercises based on the student's 's historical exercise responses. The generated exercises must adhere to the following criteria:
1. They should cover knowledge concepts relevant to the student's historical answer records and align with the student's current level of proficiency.
2. They must include clear, logical, and well-justified options.
3. They should avoid duplicating exercises or incorporating ambiguous options.
The student's historical exercise performance is provided below: (student _history).
Please ensure that the format of the generated exercises strictly conforms to the following example. the knowledge concept in a newly generated exercise overlaps with one in the historical records, retain the original name of the knowledge concept. Additionally, output the exercises strictly in the specified format, excluding any extraneous statements.
Example Format:
1. Exercise:
Sample Exercise
Options:
( ('A': 'Option 'B': 'Option 2', 'C': 'Option 3', 'D': 'Option 4')
Answer:
['C']
Concept: Enowledge_Concep
In-Context Learning (ICL)
You are an expert in generating single-choice exercises for educational purposes. Your task is to create 10 high-quality single-choice exercises based on the student's 's historical exercise responses and the provided examples. The generated exercises must adhere the following criteria:
1. They should cover knowledge concepts related to the student's 's historical answer records and correspond to the student's current level of proficiency.
2. They must include clear and logically structured options.
3. They should avoid duplicating exercises or incorporating ambiguous options.
Example exercise generation is as follows:
Student's 's performance in answering history exercises:
22

---

Content: "Data communication between different VLANs requires forwarding through
O."
Options:
"A: HHBB'
"B: 'Layer Switch',
CC: RRouerr,
'D': 'Repeater',
Answer: ['C']
is_cssrc_ee corrrcttt
Concept: K. L_ 2_Switch_____t _ Science and Technology
Content:
"Which of the following methods can be used to divide a VLAN? O)
Options:
'A': 'Based on port assignment',
"B': "Based on MAC address',
'C": 'Based on network layer (IP address)',
"D': 'TP multicast as VLAN }
Answer: ['A', 'C', 'D']
is_ _s_coreet: 0
Concept: K IP Adddress Intossatioooooonnnooaa aar_iaaanddd__cceeei Communication Enniineernnnnnnnnnngngn
Content: "What data processing method is required to ensure the correctness of data read from a distributed storage system? ()"
Options:
'A": "Compression',
'B": Multiple storage copies',
'C": 'Erasure coding'.
'D:: "Checksum data verification'
Answer: ['D']
is corrrettt: 1
Concept: K.
SeaeeeddSttrrrSSSSeS Comppteer_Scienncee TTchhology
Generated Exercises:
Exercise: "Which device can be used implement communication between
different VLANs? ()"
Options:
'A': 'HUB',
'B:: 'Layer Switch',
'C:: 'Router',
'D': 'Repeater'
Answer: ['C'] Concept_id id: K K_Layer_2_SSe Swwtchh L___SSttt_____uueeeeee Secpputer_Siennee_eee _ne_oooooToooggyy
2. Exercise: "In a distributed storage system, which method is the best choice to ensure the correctness the data read? ()"
Options:
'A': "Compression',
'B': 'Multiple storage copies',
'C": 'Erasure coding',
'D': "Checksum data verification',
Answer: ['D']
Concept:
Scompute_Sccience_a___eeeee
The student's 's historical exercise performance is provided below: bouudden_histrryll
Please ensure that the format of the generated exercises strictly conforms to the following example. If the knowledge concept in newly generated exercise overlaps with in the historical records, retain the original name of the knowledge concept. Additionally, the exercises will be output only in the specified format without any extraneous statements.
Example Format:
1. Exercise: Sample Exercise
Options: ('A': 'Option 1', 'B': 'Option 2', 'C': 'Option 3', 'D': 'Option 4')
23

---

Answer: I'Correct Answer']
Concept: Knowledge Concept
Chain-of-Thought (Co'))
You are an expert in generating single-choice exercises for educational purposes. Your task is to create 10 high-quality single-choice exercises based on the student's historical exercise responses and the provided examples. The generated exercises should be constructed following a systematic reasoning process based on the student's 's historical performance answering history exercises. Please adhere to the following procedure:
1. Analysis of Student Performance:
Identify knowledge areas where the student has demonstrated weaknesses by analyzing incorreclly answered history exercises. Assess knowledge areas where the student has shown stable mastery by considering correctly answered exercises.
2. Exercise Generation:
Include exercises that reinforce weak concepts, focusing knowledge areas where the student has previously made errors.
Incorporate exercises that test mastered concepts ensure continued proficiency.
3. Validation and Optimization:
Ensure the logical consistency of each exercise so that the exercise and its corresponding options unambiguous. Provide plausible yet non-misleading distractor options.
Optimize the clarity and readability of each exercise to facilitate comprehension.
The student's historical exercise performance is provided below: (student, _attsttttsssttr
Please ensure that the format of the generated exereises strietly conforms to the following example. If the knowledge concept in newly generated exercise overlaps with one in the historical records, retain the original name of the knowledge concept. Additionally, output only the exercises in the specified format without any extraneous statements.
Example Format:
1. Sampee Question
Options: 'A': 'Option 1', 'B': 'Option 2', 'C': 'Option 3', 'D': 'Option 4' ['Correct Answer']
Answer:
Concept:
Knowledge Concept
ICL+CoT
You are expert in generating single-choice exercises for educational purposes. Your task is to create 10 high-quality single-choice exercises based on the student's 's historical exereise responses and the provided examples.Plest Please adhere the following steps:
1. Analysis of Student Performance:
Identify the knowledge areas where the student has demonstrated weaknesses by focusing concepts answered incorrectly.
Evaluate the student's stable mastery of concepts that were answered correctly.
2. Exercise Generation:
Include exercises that reinforce weak knowledge areas (those answered incorrectly). Incorporate exercises that test the knowledge areas the student has already mastered.
3. Validation and Optimization:
Ensure the logical consistency of each exercise to guarantee clarity and accuracy. Provide plausible yet mos-laalaa distractor options. Refine the language to ensure the exercises are clear and easy to understand.
The examples provided
follows:
Student's Performance in History Exercises:
Content: "For data communication between different VLANs, data must be for- warded through O)."
24

---

Options: ('A': 'HUB', 'B'; 'Layer-2 Switch', 'C': 'Router', 'D': 'Reeeate'' |
Answer: ['C']
is correct: 0
Concept: L_Leerr-2 Switch Switch_Computer Science and Technology
2. Content: "Which of the following methods can be used to segment a Virtual LAN
(VLAN)?"
Options:
'A": SSegmenting VLANs by Port',
'B': 'Segmenting VLANs by MAC Address',
'C": "Segmenting VLANs by Network Layer (IP Address)',
'D:: 'Using IP Multicast for VLANs'
Answer: ['A', 'B", 'C', "D']
j__coreet: 0
Concept: K IP Address Inffrrmaiinn and Communication Engineering
3. Content: "In order to ensure the correctness of data read from a distributed storage system, which data processing method should be applied?" Options:
'A': 'Compression',
'B': 'Redundant Storage',
'C:: 'Erasure Coding',
'DD: "Checksum Verification'
Answer: ['D']
is_ correettt:
Concept: DiDttriitteed Storage System_ Computer Science and Technology
Generated Exercises:
Exercise: "Which of the following devices can be used facilitate communication
between different VLANs? ()"
Options: 'A': 'HUB', 'B'; 'Layer2 Switch', 'C:: 'Router', 'D': 'Repeater'
Answer: ['C']
Concept: K_Layer-2 Switch Switch_Computer Science and Technology
2. Exercise: "In distributed storage system, which method is the most effective in ensuring data accuracy during retrieval? ()"
Options: 'A'; "Compression', 'B': 'Redundant Storage', C. 'Erasure Coding",
"D::
'Checksum Verification'
Answer: ['D']
Concept: Distrriittedd Storage System_Compuue Science and Technology
The student's historical exercise performance is provided below:
(student assudent_historyy _hiistrry..............
Please ensure that the format of the generated exercises strictly conforms to the following example. If the knowledge concept in newly generated exercise overlaps with in the historical records, retain the original name of the knowledge concept. Additionally, output the exercises strictly in the specified format, excluding any extraneous statements.
Example Format:
1. Exercise: Sample Exercise
Options: | A': 'Option 1', 'B'; 'Option 2', 'C'; 'Optio 3', 'D': 'Option 4')
Answer:
['C'] Concept: Knowledge_Coom Concept
B.3 GPT-based Scoring Prompt
We introduce advanced GPT-4o to implement the evaluation for our proposed ExeGen and baselines. The evaluation prompts detailed in Figure
25

---

Task Description
You are an education expert and specialist evaluator in exercise content for (exercise type) exercises. Your task is to rigorously evaluate the generated exercises on the specific dimension: (dimension).
Dimension Definitions
1. Knowledge Relevance: Content relevance to knowledge points and teaching objectives.
2. Clarity: Expression clarity and student comprehension ease.
3. Answer Accuracy: Correctuess of answers and measurement validity.
Difficully Appropriateness: Appropriateness for students' skill level.
5. Engagement & Fun: Ability to stimulate learning interest.
6. Safety & Ethics: Compliance with ethical standards.
Evaluation Process
1. Review the 10 generated exercises based on (dimension).
Assign probability distribution P. for score [0,5]. Compute weighted score: S = o R,,
Calculate average score exercises.
Return the final scores the strict JSON format shown below.
Example Response:
("results": "final score":
Important
Strictly follow JSON format.
No additional text or explanations.
Figure 8: The prompt for GPT-based scoring.
C Real-world Deployment
To illustrate the practical utility of ExeGen, we provide two key interfaces of web application. Figure 10 illlustrate the user interfaces tailored for students and teachers, respectively. On the student-oriented interface (Figure D... the right-hand side comprises a configuration panel, which includes modules for agent selection, base LLM selection, and parameter adjustment. The upper-left area features a visualized interaction panel that displays the dialogue process among AI Agents, thereby enhancing the transparency of exercise generation. The lower-left area presents the final generated exercises. In contrast, the teacher-oriented interface (Figure T0) removes the Knowledge Perceiver agent from the agent selection panel due instructional use. Additionally, it incorporates an explicit input form the lower-left corner, allowing teachers to manually specify attributes such difficulty level, language, course, and knowledge concepts.
To balance the generation quality and response efficiency, both interfaces adopt a modular agent configuration, wherein users may selectively deactivate specific agents they consider non-essential, thereby expediting the overall exercise generation process. Besides, the exercise generation results support post-generation modifications and multi-format export functionality (TXT/PDF/JSON), enabling flexible adaptation to pedagogical requirements.
D Limitations and Future Work
Although ExeGen shows strong theoretical and empirical performance, there are several limitations that highlight significant opportunities for improvement. First, the used domain knowledge graph in ExeGen from the MOOCCubeX dataset lacks some critical information, such as prerequisite relations between concepts. Our analysis reveals that only 0.063725% concept pairs MOOCCubeX are annotated with prerequisite relations, limiting the ability of ExeGen to model knowledge dependencies. In the future, we plan to explore zero-shot few-shot knowledge graph completion techniques to enrich the knowledge graph and enhance recommendation quality. Second, ExeGen depends
26

---

Agent Visualization Window
Agent vodg
Munage
Hed agnd agnn grraai
Diaccia Ganwerror
Colcaiion Copart
Put be eseeue fral dnigree roued enploternes and folew poothed cu forme
LLM Selection Area
Liake
DeepSeer
captent io vere dioto centor seci thone to "oong partil
"seeei vnter fuud muntioo Thie malkking tycee Mil
Oven
OPS4)
Exercise Generation Results (Editable Window)
Parameter Settings Area
shu nahoo uoteeen
Compcn Teiinday
Exeecises
Figure 9: The core interface of ExeGen's web application for students.
Agent Interaction Visualization Window
Intelligent Agent Selection Area
Genaradion
Cancta Generic E
Soleeos Thays comprr bade Ting muserid Thand bal
Compoce Sceeoseeeehheeee
Cratartor Expera
Manage
LLM Selection Area
to unginall Merteteue hoa malertie
Phes tigu turnc cheare page corwaccs sabunang parcass acsaaas "olat
X LLnk
DerpSoc
Oren
Exercise Generation Results (Editable Window)
Parameter Settings Area
Pagee
Denerate Enarcises
Buttor
Figure 10: The core interface of ExeGen's 's web application for teachers.
LLMs to reason over students' historical exercise responses, but LLMs may struggle with long interaction histories, leading to unstable contextual reasoning. We plan to investigate methods for improving long-context modeling in this setting. Finally, although we conduct human evaluations from both student and teacher perspectives as well compare the evaluation results with GPT-based scores, the scale of manual evaluation is relatively small. We plan to extend the human evaluation experiments with large-scale online A/B testing using our deployed application in real educational environments.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contributions and scope of the paper clearly included in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitation discussed in the Appendix D.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper provides detailed descriptions of the experimental setup, including dataset specifications, baseline selection, evaluation protocol, and implementation details, in Section 5 and Appendix A. Besides, we release the code and data to ease the reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
29

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The used data and code are opened.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
praips.cc/ubblicgguuueee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper describes all relevant experimental details Section 5 and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports appropriate information about the statistical significance of the experiments in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
30

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
The paper uses the OpenAl API for experiments and reports the cost in Appendix A.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics hattps://neutt httpp//rwwrpp..pcppceeeeeeeeeeee httpp/pc....ppp.tpttttccccccccaa
Answer: [Yes]
Justification:
The research ensures that ethical considerations are addressed and integrated into the research's design and execution. And the authors ensure anonymity is preserved.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential impacts that our study will bring in Section 1.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
31

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not involve the release of data models that pose high risk for misuse, thus no safeguards are necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original paper or attached the link to the existing assets used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
32

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have attached the introduction of how the code and the license in the anonymized code repository.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
33

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
The paper describes the usage of LLMs in Section and Appendix
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp:r:////upp...../ h0tps//eecccoooeeeeee/////////// for what should or should be described.
34


Reviews:
[{"review_id": "Wy8Vspcgnp", "reviewer": "Reviewer_R2cu", "summary": "This work focuses on adaptive learning and proposes a novel task called Knowledge-aware Exercise Generative Recommendation (KEGR). To achieve good performance, the authors introduce an adaptive multi-agent cooperation framework, ExeGen, and explore techniques such as prompting and adversarial refinement. Experiments demonstrate the effects of the proposed method.", "questions": "Questions:\n\n1. Is the Human-simulated Knowledge Perception Mechanism reasonable, given that different students have different knowledge bases?\n\n2. Can this framework handle a large amount of historical response data?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The authors effectively explain the motivation for this work in the introduction and clearly define the KEGR problem, including Knowledge Tracing (KT) and Exercise Recommendation (ER).\n\n2. The concept of building a Human-simulated Knowledge Perception Mechanism is interesting.\n\nWeaknesses:\n\n1. The proposed framework heavily relies on data/analysis generated by GPT, which may introduce hallucination issues, even with techniques like prompting and adversarial refinement.\n\n2. Most evaluation metrics are based on questionnaires filled out by GPT/humans, not convincing.\n\n3. The correctness of generated knowledge state descriptions is crucial for high-quality exercise generation, but no evaluations on this aspect are provided.\n\n4. The dataset's diversity is limited to only one Chinese dataset. More datasets are needed to demonstrate the framework’s generalization ability.", "was_revised": true, "final_justification": "I have decided to slightly increase the score based on the additional experimental results.", "rebuttal": "**Response to W1:**\n\nThank you for raising this important concern. We acknowledge the hallucination risks inherent in LLM-generated content. However, our goal is to leverage the strengths of LLMs for large-scale personalized exercise generation while actively mitigating hallucination issues, rather than aiming for complete elimination. ExeGen is built on LLMs due to their strong reasoning ability, cross-disciplinary versatility, and language generation capability, which enable accurate inference of students’ knowledge states and personalized exercise generation. Prior work [1,2] has shown that LLMs can perform reliably in educational knowledge-intensive tasks with structured prompting and multi-step reasoning. Thus, leveraging ExeGen for personalized exercise generation is feasible.\n\nTo eliminate hallucination, ExeGen incorporates three technical innovations and one practical safeguard: 1) A human-simulated knowledge perception mechanism enforces step-by-step reasoning of “knowledge retrieval → analysis → exercise solving”, improving transparency and reducing single-step generation errors. 2) We integrate domain knowledge via knowledge-graph-enhanced prompting and dual-matching of high-quality demonstrations to ensure factual grounding. 3) An exercise generative-adversarial mechanism introduces expert agents to evaluate and refine exercises along three dimensions (linguistic fluency, coverage of erroneous knowledge concepts, and correctness) through adversarial feedback. 4) As shown in Appendix C, the application interface allows teachers to edit generated exercises, ensuring human oversight before delivery.\n\nIn summary, ExeGen combines LLM strengths with structured control and teacher-in-the-loop validation to reduce hallucination risks. We appreciate your feedback and will explore advanced hallucination mitigation strategies such as reinforcement learning with human feedback and prompt tuning in future work.\n\n[1] SocraticLM: Exploring Socratic Personalized Teaching with Large Language Models, NeurIPS 2024.\n\n[2] What Makes In-context Learning Effective for Mathematical Reasoning, ICML 2025.\n\n**Response to W2:**\n\nThank you for your valuable comments. We would like to clarify the design rationale and rationality of our evaluation protocol.\n\nTraditional metrics, such as BLUE and Accuracy, fall short in capturing essential aspects of open-ended educational generation tasks, including personalization, pedagogical appropriateness, and knowledge alignment. Recent NeurIPS [1], ICLR [2], and ACL [3] papers have increasingly adopted hybrid evaluation protocols that combine human judgment and LLM-based scoring, reflecting the emerging consensus and rationality of these evaluation protocols for open-ended generation tasks.\n\nTo ensure a reliable evaluation, we design a hybrid evaluation protocol covering four perspectives: 1) Classical statistical metrics (Recall and HR) assess whether generated exercises align with students’ weak knowledge concepts. 2) GPT-based scoring evaluates each exercise along six dimensions. 3) Human evaluations from both teachers and students serve as a cross-check for GPT-based scoring. 4) As shown in Appendix C, we deploy ExeGen in real-world college scenarios, supporting teacher-in-the-loop review and editable outputs.\n\nIn summary, GPT and human ratings are only part of our evaluation protocol. As also acknowledged by reviewers 7wBG, zZz2, and R2cu, our evaluation protocol offers a reasonable and trustworthy approach in the absence of standardized metrics for LLM-based educational generation tasks. We appreciate your comments and will continue refining our evaluation methods.\n\n[1] SocraticLM: Exploring Socratic Personalized Teaching with Large Language Models, NeurIPS 2024.\n\n[2] The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning, ICLR 2024.\n\n[3] From Objectives to Questions: A Planning-based Framework for Educational Mathematical Question Generation, ACL 2025.\n\n**Response to W3:**\n\nThank you for your valuable comments. The correctness of generated knowledge state descriptions is crucial for high-quality exercise generation. Our framework explicitly addresses this through the Knowledge Perceiver agent, which integrates a human-simulated knowledge perception mechanism to emulate the reasoning process of human exercise-solving, thereby accurately inferring the personalized knowledge states of students. To evaluate the effectiveness, we conduct comprehensive experiments from multiple perspectives:\n\n**Ablation study** (§5.3, Table 2): Removing the Knowledge Perceiver (w/o. KP) and the human-simulated knowledge perception mechanism (w/o. HKP) results in the most significant performance drops, confirming their central role and effectiveness for exercise generation.\n\n**Human expert evaluation** (Appendix§A.5.3, Figure 7): Teachers rate the generated knowledge state descriptions with an average Knowledge Relevance (KR) score of 4.47 out of 5, indicating strong alignment with students’ actual response performance.\n\n**GPT-based scoring** (§5.3, Table 1): The KR score rated by GPT-4o averages 4.60, further supporting the consistency between generated knowledge states and exercises.\n\n**Case studies** (§5.5, Figure 5): A real case illustrates that ExeGen accurately identifies students’ underlying conceptual weakness from their historical responses.\n\nMoreover, we additionally evaluate the quality of knowledge state descriptions. This includes traditional metrics (Recall, Language Fluency) and GPT-based scoring across four dimensions (Clarity, Safety and Ethics, Consistency, and Granularity). The results (see table below) further demonstrate the accuracy, reliability, and fine-grained quality of our knowledge state perception.\n|Methods|Recall$\\uparrow$|Language Fluency$\\downarrow$|Clarity$\\uparrow$|Safety and Ethics$\\uparrow$|Consistency$\\uparrow$| Granularity$\\uparrow$|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|ExeGen|0.84|45.18|4.31|4.47|4.37|4.30|\n\n**Response to W4:**\n\nThank you for your constructive comments. To demonstrate the generalization of ExeGen, we conduct additional experiments on MathDial, an English dialogue-based math dataset with 2,848 test samples. Due to time and space constraints, we only evaluate on single-choice exercise generation. The experimental results show that ExeGen consistently outperforms all baselines, demonstrating its strong generalization ability.\n|Methods|KR|Clarity|AA|DA|EF|SE|HR|Recall|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Zero_shot|3.60|3.45|3.50|3.30|3.00|4.20|0.08|0.15|\n|ICL|4.05|4.15|4.00|4.00|3.90|4.10|0.30|0.40|\n|CoT|4.10|4.20|4.00|3.95|3.75|3.90|0.28|0.38|\n|ICL+CoT|4.20|4.30|4.25|4.15|4.10|4.25|0.20|0.45|\n|ExeGen|4.65|4.50|4.55|4.60|4.35|4.80|0.60|0.65|\n\n**Response to Q1:**\n\nThank you for raising this important question. We fully understand the concern regarding the adaptability of the mechanism to students with diverse knowledge bases. Below, we clarify its motivation, implementation, and experimental validation.\n\nThis mechanism addresses two key limitations in knowledge tracing and recommendation systems: insufficient semantic mining and poor interpretability. With recent advances in LLMs, it becomes feasible to infer a student’s personalized knowledge state from historical responses and express it in natural language descriptions for improving interpretability. However, LLMs face two challenges in knowledge state perception: lack of domain-specific knowledge and vulnerability to hallucinations during complex reasoning. To address these challenges, this mechanism incorporates a domain-specific knowledge graph to enhance factual grounding and aligns LLM reasoning with a multi-step cognitive process. This design improves reliability, transparency, and adaptability in knowledge state modeling.\n\nThis mechanism consists of two stages: **Demonstration construction** builds a repository of cognitive reasoning paths from real student data, each covering knowledge retrieval, analysis, and exercise-solving. These demonstrations cover various levels of knowledge mastery, which serve as experience templates rather than student-specific examples. **Knowledge state generation** uses a student’s historical responses to retrieve relevant demonstrations from the constructed repository as contextual guidance. This allows the model to adaptively infer personalized knowledge states based on individual learning histories.\n\nFinally, extensive experiments (see our response to W3) confirm the effectiveness of this mechanism in accurately modeling diverse student knowledge states.\n\n**Response to Q2:**\n\nThank you for highlighting this important concern. Handling large-scale historical response data is critical for real-world educational applications. We clarify this question from both the architectural design and empirical validation.\n\nArchitecturally, ExeGen adopts a multi-agent design that decomposes the knowledge-aware exercise generative recommendation task into modular sub-tasks, including recommendation management, knowledge perception, exercise generation, and quality refinement. Each agent processes only subtask-relevant information, which reduces both information overhead and prevents context overflow. This modularization design improves both scalability and interpretability, especially when handling complex, personalized reasoning tasks.\n\nIn practice, we evaluate ExeGen on the large-scale real-world education dataset MOOCCubeX, where students have an average of 1,695 tokens in historical responses (max: 1,926 tokens). The average context lengths for the knowledge perceiver, exercise generator, and quality evaluation expert are 2,244, 922, and 2,483 tokens, respectively. These token lengths are all well within the 128k token limit of GPT-4o, which serves as our backbone LLM.\n\nIn summary, ExeGen can effectively handle large-scale input through modular processing and has demonstrated practical scalability in real-world datasets."}, {"review_id": "A2muJ7MLLJ", "reviewer": "Reviewer_zZz2", "summary": "This paper proposes Knowledge-aware Exercise Generative Recommendation (KEGR), a task that unifies knowledge state diagnosis and personalized exercise generation. To achieve this, the paper develops ExeGen, a multi-agent framework powered by LLMs, where four specialized agents collaborate in a closed-loop pipeline: Recommendation Manager, Knowledge Perceiver, Exercise Generator, and Quality Evaluation Experts. Furthermore, key techniques also include a human-like knowledge perception mechanism using knowledge graphs and in-context learning, and an adversarial refinement process to improve exercise quality. Experiments on real-world datasets and deployment in college settings show ExeGen’s superiority over existing baselines, validating its potential for adaptive and interpretable learning support.", "questions": "1. Could the paper clarify what specific architectural or procedural innovations distinguish their multi-agent framework from simply coordinating ICL and CoT within standard LLM pipelines? What novel agent-specific interactions or functionalities are introduced beyond existing prompting paradigms?\n\n2. Have the paper conducted or considered any longitudinal studies to measure actual student learning gains after engaging with the generated exercises? If not, how do they justify the use of GPT-based and human scoring as sufficient proxies for real educational effectiveness?\n\n3. Given that the Knowledge Perceiver is designed to model student knowledge states, why is there no quantitative comparison with established knowledge tracing models such as BKT, DKT, or GKT? \n\n4. Many existing frameworks like ReAct and Reflexion have demonstrated strong performance in agent-based reasoning tasks. Could the authors provide empirical comparisons with these frameworks or explain in detail why such approaches are unsuitable for the KEGR task?\n\n5. How exactly are the Human-simulated Knowledge Perception Mechanism and the Exercise Generative-Adversarial Mechanism integrated into the four-agent architecture? Which agent(s) are responsible for executing these mechanisms, and how are they invoked in the workflow?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper introduces KEGR, a meaningful task that integrates knowledge tracing and exercise recommendation into a unified, closed-loop adaptive learning system.\n\n2. ExeGen's use of four specialized LLM-powered agents—each responsible for a distinct subtask—ensures transparency and modularity in handling complex reasoning and generation tasks. Furthermore, Human-Simulated Knowledge Perception and Adversarial Quality Refinement strengthen the above agent’s ability, contributing to a more robust workflows.\n\n3. The system is evaluated comprehensively through a hybrid approach that includes GPT-based metrics, statistical scores, student questionnaires, and teacher expert assessments, which is not only tested offline but also deployed in a real college environment, providing both quantitative and qualitative validation and also supporting its practical utility and readiness for real-world application.\n\nWeakness:\n\n1. The proposed agent-based framework lacks substantial technical novelty. It heavily relies on existing techniques such as In-Context Learning (ICL) and Chain-of-Thought (CoT), and does not demonstrate significant innovation in the design of the agent architecture itself.\n\n2. The current evaluation focuses primarily on the quality of the generated exercises, assessed mainly through GPT-based scoring and human judgments. However, these metrics are inherently subjective and do not reflect the core objective of exercise recommendation—namely, the actual improvement in students' knowledge and learning outcomes as measured by their performance after doing exercised generated by this framework.\n\n3. While the motivation of the agent design is to address limitations in both knowledge tracing and exercise recommendation, the evaluation fails to provide a detailed comparison against traditional models from these domains. For instance, the Knowledge Perceiver module is intended to trace student knowledge states, yet its performance is not compared against standard knowledge tracing models, making it difficult to assess its effectiveness.\n\n4. Although the paper is the first to propose the KEGR task and solve it via an agent-based approach, there are already many well-established agent frameworks such as ReAct, Reflexion, etc. Including empirical comparisons with these frameworks, or clearly explaining why they are not suitable for KEGR, would significantly strengthen the contribution.\n\n5. The paper’s presentation is somewhat confusing. For example, in addition to the four core agent’s components, the authors introduce two mechanisms—Human-simulated Knowledge Perception and Exercise Generative-Adversarial mechanisms—but do not clearly specify how these techniques are integrated into which component of the framework. Clarifying their roles and placement within the overall framework would enhance clarity.", "was_revised": true, "final_justification": "The concerns are addressed, and the score is raised. Thanks for the efforts.", "rebuttal": "**Response to W1&Q1:**\n\nThank you for the thoughtful feedback. We respectfully clarify that ExeGen is not a simple combination of existing techniques like ICL or CoT, but a task-driven, mechanism-enhanced multi-agent framework designed for a novel educational application task. Our technical contributions span task reasoning mechanisms, task formulation, and framework design.\n\nFor reasoning mechanisms, we introduce two innovations that go beyond standard ICL or CoT. First, the human-simulated knowledge perception mechanism guides LLMs to emulate fine-grained cognitive reasoning over student historical responses. It incorporates domain-specific knowledge graphs, dual-matching retrieval strategies, and multi-step human-simulated recognition reasoning to produce accurate knowledge state descriptions, which surpasses standard ICL’s static prompt-based strategy and CoT’s single-chain reasoning. Second, the exercise generative-adversarial mechanism creates an iterative feedback loop between the exercise generator and evaluation expert agents, enabling multi-dimensional educational critique and refinement, which improves generation beyond one-pass Cot/ICL.\n\nFor task formulation, we define KEGR as a new task that reformulates and unifies Knowledge Tracing (KT) and Exercise Recommendation (ER) into a generation-based setting. Unlike these binary classification (KT) or matching (ER) tasks, KEGR aims to dynamically perceive students’ knowledge states from historical exercise responses, and subsequently generate tailored exercises based on the inferred knowledge states.\n\nFor framework design, ExeGen is a modular architecture with four collaborative agents, responsible for global supervision, knowledge state perception, exercise generation, and quality refinement, respectively. This setup supports complex-task-oriented adaptive reasoning, interpretability, and future extensibility, beyond the combination of ICL and CoT.\n\nFinally, as shown in Table 2 (§5.3), ExeGen significantly outperforms strong baselines including CoT, ICL, and hybrid methods, confirming its improved reasoning and generation capabilities.\n\nIn summary, ExeGen contributes new technical mechanisms and a task-specific multi-agent architecture that meaningfully extend LLM reasoning for personalized education.\n\n**Response to W2&Q2:**\n\nThank you for your insightful comment. We fully agree that the ultimate goal of educational AI systems is to improve student learning outcomes. Our current work focuses on a key prerequisite: generating high-quality, personalized exercises based on students’ historical responses. This forms the foundation for long-term learning gains and is a key part of our team’s long-term research plan.\n\nTo this end, we emphasize the evaluation of the quality and personalization of the generated exercises. We design a multi-dimensional protocol covering four perspectives: 1) Classical statistical metrics (Recall and HR) assess whether generated exercises align with students’ weak knowledge concepts. 2) GPT-based scoring evaluates each exercise along six dimensions. 3) Human evaluations from both teachers and students serve as a cross-check for GPT-based scoring. 4) We deploy ExeGen in real-world college scenarios, supporting teacher-in-the-loop review and editable outputs. Results consistently show that ExeGen effectively generates high-quality exercises, which are closely aligned with students’ knowledge states.\n\nWe appreciate the suggestion to conduct longitudinal studies. We have deployed ExeGen in a real college environment (see Appendix C) and are actively collecting student data. While direct learning gain evaluation is important, it is inherently long-term and influenced by many variables including instructional context, external interventions, and student engagement. As a step forward, we are conducting process-level evaluations such as in-class student behavior analysis.\n\nWe plan to extend this into systematic studies on learning gains in future work. Thank you again for your thoughtful feedback and support.\n\n**Response to W3&Q3:**\n\nThank you for your valuable comments. We agree that validating the effectiveness of the Knowledge Perceiver (KP) module is essential and appreciate the opportunity to clarify its design objectives and experimental evaluation.\n\nUnlike traditional Knowledge Tracing (KT) models such as DKT, which perform binary classification of next-exercise response correctness, the KP module aims to generate interpretable natural language descriptions of students’ knowledge states. These informative descriptions serve as conditional input for downstream personalized exercise generation, shifting the focus from classification of the KT task to generation and explanation. As noted in lines 45-47 and 125-126 of the manuscript, this task is completely different from the KT task, and this design bridges the gap between knowledge state modeling and personalized exercise generation. To evaluate its effectiveness, we adopt a multi-faceted evaluation strategy:\n\n**Ablation study** (§5.3, Table 2): Removing the Knowledge Perceiver (w/o. KP) results in the most significant performance drop, confirming its central role and effectiveness for exercise generation.\n\n**Human expert evaluation** (Appendix§A.5.3, Figure 7): Teachers rate the generated knowledge state descriptions with an average Knowledge Relevance (KR) score of 4.47 out of 5, indicating strong alignment with students’ actual response performance.\n\n**GPT-based scoring** (§5.3, Table 1): The KR score rated by GPT-4o averages 4.60, further supporting the consistency between generated knowledge states and exercises.\n\n**Case studies** (§5.5, Figure 5): A real case illustrates that ExeGen accurately identifies students’ underlying conceptual weakness from their historical responses.\n\nAdditionally, we conduct a new experiment focusing specifically on the generated knowledge state descriptions. Results across traditional metrics (Recall, Language Fluency) and GPT-based scoring metrics (Clarity, Safety and Ethics, Consistency, and Granularity) confirm the precision and reliability of the KP outputs.\n|Methods|Recall$\\uparrow$|Language Fluency$\\downarrow$|Clarity$\\uparrow$|Safety and Ethics$\\uparrow$|Consistency$\\uparrow$| Granularity$\\uparrow$|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|ExeGen|0.84|45.18|4.31|4.47|4.37|4.30|\n\n**Response to W4&Q4:**\n\nThank you for recognizing the novelty of the KEGR task and for your professional suggestion. Comparing with existing agent frameworks helps clarify our technical contributions.\n\nKEGR requires generating personalized exercises based on the inferred knowledge states from historical responses in real-world educational settings. This task involves a structured pipeline with multiple sub-tasks, including knowledge perception, exercise generation, quality refinement, and recommendation. To address this, ExeGen adopts a domain-specific multi-agent architecture with four specialized agents: Knowledge Perceiver, Exercise Generator, Quality Evaluation Expert, and Recommendation Manager.\n\nIn contrast, frameworks like ReAct, Reflexion, and AutoGen are primarily designed for open-domain reasoning or autonomous planning tasks such as QA. These frameworks focus on general-purpose reasoning, often using trial-and-error or environment-driven feedback, without explicit support for personalization or domain-specific evaluation. To illustrate the distinction, we summarize key differences in the table below:\n|Aspect|ExeGen|ReAct|Reflexion|AutoGen|\n|:-:|:-:|:-:|:-:|:-:|\n|Task type|Structured educational resource generation (KEGR)|Open-domain reasoning or autonomous planning tasks|Open-domain reasoning or autonomous planning tasks|Open-domain reasoning or autonomous planning tasks|\n|Architecture|Multi-agent, task-specific|Single-agent|Single-agent|Multi-agent|\n|Reasoning|Adaptive closed-loop workflow with knowledge guidance|Trial-and-error|Trial-and-error reasoning with environmental feedback|Autonomous dialogue-based coordination|\n|Personalization|Student-specific knowledge modeling|Not addressed|Not addressed|Not addressed|\n|Verification|Multi-dimensional educational evaluation for refinement|None|Environment-driven feedback for reflection|No explicit mechanism|\n\nAs shown, ExeGen is tailored to the task of KEGR, integrating structured reasoning, personalization, and rigorous quality control, which are not supported by general-purpose frameworks.\n\n**Response to W5&Q5:**\n\nThank you for highlighting this clarity issue. We now clarify how the two mechanisms integrate with the agent components of the ExeGen.\n\nExeGen consists of four core agents: Recommendation Manager, Knowledge Perceiver, Exercise Generator, and Quality Evaluation Expert. These agents collaborate to deliver personalized, high-quality exercise recommendations. The two mechanisms are integrated as follows:\n\nThe human-simulated knowledge perception mechanism is embedded within the Knowledge Perceiver agent. It serves as the core reasoning engine for modeling students’ knowledge states. This mechanism guides the LLM through a human-simulated cognitive process: knowledge retrieval→analysis→exercise answering, to infer interpretable and fine-grained knowledge states. In the original manuscript, we outline this integration in lines 175-180 of §4.1, and detail this mechanism in §4.2.\n\nThe exercise generative-adversarial mechanism operates between the Exercise Generator agent and Quality Evaluation Expert agents. It forms an iterative feedback loop in which the generator creates exercises and the expert agents evaluate them along key educational dimensions. The generator then refines outputs based on the expert agent feedback. In the original manuscript, this integration is described in lines 190-196 of §4.1 and elaborated in §4.3.\n\nWhile both mechanisms are discussed in the original manuscript, we agree that their connections to specific agents could be more explicitly illustrated."}, {"review_id": "PMq5t4qGRa", "reviewer": "Reviewer_7wBG", "summary": "This paper proposes a novel and practical task _Knowledge-aware Exercise Generative Recommendation_, and introduces a LLM-based multi-agent framework. Its thoughtful design and real-world validation demonstrate strong potential for adaptive learning. The real-world deployment supports the claims of effectiveness and superiority.", "questions": "1. The manuscript acknowledges that LLMs may hallucinate. To address this, it introduces a human-simulated knowledge perception mechanism based on a domain knowledge graph, involving three steps: relevant knowledge retrieval, analysis, exercise solving. It is recommended to evaluate this pipeline experimentally and identify subject areas where it may not perform well.\n2. The experimental setup lacks clarity, especially regarding the subject domains (e.g., math, science). Since knowledge structures vary across subjects, it's important to test the method’s effectiveness in different domains and discuss whether the pipeline needs subject-specific adaptations.\n3. The completeness of the knowledge graph is crucial for accurate knowledge state estimation. A more thorough analysis of its coverage (especially missing prerequisite links), is needed to assess its impact on system performance.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. Bridges knowledge tracing and exercise recommendation, forming a closed-loop adaptive system.\n2. Four LLM agents (Manager, Perceiver, Generator, Evaluator) work in a feedback loop, effectively handling the KEGR task.\n3. The real-world deployment demonstrates its effectiveness.\n\nWeakness:\n1. While the proposed human-simulated knowledge perception mechanism aims to reduce LLM hallucination through a three-step process (knowledge retrieval, analysis, exercise solving), its effectiveness has not been independently validated.\n2. The current experiments focus on a single subject domain, but this is not clearly stated. Since knowledge structures differ significantly between disciplines, it remains unclear how well the proposed method generalizes across domains or whether subject-specific adjustments are needed.\n3. The domain knowledge graph used in this study lacks completeness, particularly in prerequisite relationships between concepts. This limitation may affect the accuracy of knowledge state modeling and the quality of generated exercises. A more detailed analysis of the knowledge graph’s coverage and its impact on system performance is needed.", "was_revised": true, "final_justification": "1. The authors performed new tests to address the concerns about the impact of incomplete knowledge graphs.\n2. The authors clarified that the knowledge graph acts as auxiliary guidance and the model is not fundamentally dependent on its completeness.\n3. The rebuttal reaffirmed that the main contributions of the paper, such as the KEGR task, the multi-agent framework, and the enhancement mechanisms, are robust and effective regardless of the knowledge graph's completeness.", "rebuttal": "**Response to W1&Q1:**\n\nThank you for the recognition of our work and the thoughtful comments. We agree that validating the effectiveness of the human-simulated knowledge perception mechanism is essential.  This mechanism aims to emulate human exercise-solving processes: knowledge retrieval→analysis→exercise answering, to accurately perceive the personalized knowledge states of students. In this paper, we have conducted multiple experimental evidence to support its effectiveness:\n\n**Ablation study** (§5.3, Table 2): Removing the human-simulated knowledge perception mechanism (w/o. HKP) results in a significant performance drop, confirming its critical role and effectiveness for knowledge state perception and personalized exercise generation.\n\n**Human expert evaluation** (Appendix§A.5.3, Figure 7): Teachers rate the generated knowledge state descriptions with an average Knowledge Relevance (KR) score of 4.47 out of 5, indicating strong alignment with students’ actual response performance.\n\n**GPT-based scoring** (§5.3, Table 1): The KR score rated by GPT-4o averages 4.60, further supporting the consistency between generated knowledge states and exercises.\n\n**Case studies** (§5.5, Figure 5): A real case illustrates that ExeGen accurately identifies students’ underlying conceptual weakness from their historical responses.\n\nAdditionally, we conduct a new experiment focusing specifically on the generated knowledge state descriptions. Results across traditional metrics (Recall, Language Fluency) and GPT-based scoring metrics (Clarity, Safety and Ethics, Consistency, and Granularity) confirm the precision and reliability of the knowledge state modeling.\n|Methods|Recall$\\uparrow$|Language Fluency$\\downarrow$|Clarity$\\uparrow$|Safety and Ethics$\\uparrow$|Consistency$\\uparrow$| Granularity$\\uparrow$|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|ExeGen|0.84|45.18|4.31|4.47|4.37|4.30|\n\nThese results offer quantitative evidence of the mechanism’s effectiveness. We appreciate your comment, which helps us strengthen the effectiveness demonstration of our work.\n\n**Response to W2&Q2:**\n\nThank you for raising this important point. We clarify that although our main experiments focus on the computer science domain, the core components of ExeGen, including the multi-agent cooperation framework, human-simulated knowledge perception mechanism, and exercise generative-adversarial mechanism, are designed to be domain-agnostic. ExeGen relies on three transferable elements: 1) domain-specific knowledge graphs; 2) human-cognition-inspired multi-step reasoning guided by demonstrations; and 3) iterative multi-dimensional quality optimization. These elements can be adapted to other subjects with well-defined knowledge structures, such as mathematics or physics. To assess generalization, we conduct additional experiments on the mathematics domain using the MOOCCubeX dataset. Due to time and space constraints, we only evaluate on single-choice exercise generation. As shown in the table below, ExeGen consistently outperforms all baselines across all metrics, demonstrating its strong generalization.\n|Methods|KR|Clarity|AA|DA|EF|SE|HR|Recall|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Zero_shot|3.68|3.53|3.6|3.38|3.12|4.12|0.11|0.18|\n|ICL|4.02|4.07|4.01|3.92|3.95|4.05|0.32|0.41|\n|CoT|4.15|4.18|4.06|4.00|3.80|4.00|0.30|0.39|\n|ICL+CoT|4.30|4.33|4.31|4.25|4.15|4.30|0.22|0.49|\n|ExeGen|4.72|4.62|4.65|4.70|4.50|4.85|0.62|0.70|\n\nMoreover, we conduct additional experiments on MathDial, an English dialogue-based math dataset with 2,848 test samples. The experimental table below shows that ExeGen consistently outperforms all baselines across all metrics, further demonstrating its strong generalization ability.\n|Methods|KR|Clarity|AA|DA|EF|SE|HR|Recall|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Zero_shot|3.60|3.45|3.50|3.30|3.00|4.20|0.08|0.15|\n|ICL|4.05|4.15|4.00|4.00|3.90|4.10|0.30|0.40|\n|CoT|4.10|4.20|4.00|3.95|3.75|3.90|0.28|0.38|\n|ICL+CoT|4.20|4.30|4.25|4.15|4.10|4.25|0.20|0.45|\n|ExeGen|4.65|4.50|4.55|4.60|4.35|4.80|0.60|0.65|\n\n**Response to W3&Q3:**\n\nThank you for your insightful suggestions. We agree that the completeness of the domain-specific knowledge graph, particularly with regard to prerequisite relationships among concepts, might affect the quality of knowledge state perception and exercise generation. However, in ExeGen, the knowledge graph plays an auxiliary supporting role, serving as factual knowledge guidance to help the human-simulated knowledge perception mechanism mitigate the hallucinations of LLMs and improve inference reliability. It is not the core component for knowledge state modeling or exercise generation.\n\nParticularly, we appreciate your valuable opinion, which is highly relevant to our ongoing work on LLM-assisted knowledge graph completion. We hypothesize that LLMs can compensate for incomplete graphs by leveraging internalized domain knowledge. To explore this, we conduct a preliminary study on MOOCCubeX by randomly masking a subset of prerequisite triples and prompting GPT-3.5-turbo to infer the masked relations. As shown below, GPT-3.5-turbo significantly outperforms traditional knowledge graph completion methods, suggesting strong potential for improving knowledge graph coverage.\n|Metrics|TransE|DistMult|ComplEx|CP|InGram|GPT-3.5-turbo|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|ACC|29.38%|42.76%|47.58|42.10%|52.62%|62.37%|\n|F1|32.35%|49.71%|53.22%|56.72%|49.03%|63.63%|\n\nWe are integrating this insight into future versions of ExeGen by adding a dedicated knowledge graph completion and refinement module. Your feedback not only confirms the value of our current design but also points to an exciting direction for further enhancement."}, {"review_id": "6ruaz2eegH", "reviewer": "Reviewer_G5ta", "summary": "This paper presents ExeGen, a novel multi-agent framework for Knowledge-aware Exercise Generative Recommendation (KEGR). By coordinating LLM-powered agents for knowledge perception, exercise generation, and quality refinement, ExeGen provides personalized and interpretable learning support. Extensive experiments and real-world deployment validate its effectiveness in adaptive education.", "questions": "1. In this work, all exercises are generated by LLMs. Although a corresponding evaluation expert system is designed, it still cannot fundamentally address the issue of ensuring that the generated exercises fully comply with educational standards, especially for complex problems.\n2. In addition to methods like CoT, I believe it would be beneficial to incorporate approaches such as Self-Consistency [1] and Best-of-N [2] to further validate the superiority of the proposed method.\n3. The current framework only supports objective exercise types (single choice, multiple choice, and judgment), and has not yet addressed other types of exercise.\n\n[1]Self-consistency improves chain of thought reasoning in language models. ICLR 2023.\n\n[2]Let’s verify step by step. ICLR 2024.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed KEGR framework dynamically infers students’ knowledge states from their past exercise responses and customizes the generation of new exercises, making the design more aligned with real-world learning scenarios.\n2. The incorporation of a human-simulated knowledge perception mechanism and an adversarial refinement loop ensures both the interpretability of the learner's knowledge state and the high quality of the generated exercises.\n3. The proposed method has been deployed in real school settings, with application screenshots provided, which strongly demonstrates the practical validity of the approach.\n\nWeaknesses:\n1. To the best of my knowledge, even the most powerful large language models still suffer from hallucination issues. Since all the exercises in the paper are generated by LLMs, the proposed evaluation expert system—though carefully designed—cannot fundamentally guarantee that the generated exercises fully comply with educational standards.\n2. Due to the lack of directly comparable baselines, the authors adopt four representative methods. It may be worth considering the inclusion of some advanced LLM reasoning variants (e.g., Tree-of-Thoughts) to further demonstrate the effectiveness of the proposed approach.\n3. The framework currently supports only objective exercise types (single choice, multiple choice, judgment), omitting more complex subjective or open-ended tasks that are critical in higher-order learning assessment.", "was_revised": true, "final_justification": "Thanks author's response. I will maintain my original positive score.", "rebuttal": "**Response to W1&Q1:**\n\nThank you for your insightful comments. We fully agree that hallucination remains a key limitation of LLMs particularly for educational applications. We would like to clarify our design motivation and hallucination mitigation strategies.\n\nFirst, we build our ExeGen framework on LLMs such as GPT due to their strong reasoning capabilities, cross-disciplinary versatility, and language generation diversity. These strengths enable ExeGen to infer students’ knowledge states from historical responses and generate tailored exercises at scale. Recent studies [1,2] have shown that, when paired with carefully designed prompts and multi-step reasoning, LLMs can be successfully applied to education scenarios, such as personalized tutoring. Therefore, leveraging ExeGen for personalized exercise generation is feasible and important to achieve large-scale personalized teaching for real-world applications.\n\nSecond, we recognize that generating high-quality, knowledge-aware, and personalized exercises remains a non-trivial task. Simple prompting often leads to shallow, unstructured, or inaccurate content. To address this, we introduce several novel designs in our proposed ExeGen, including a multi-agent cooperation framework, a human-simulated knowledge perception mechanism, and an exercise generative-adversarial mechanism, to improve reasoning coherence and educational alignment.\n\nThird, to eliminate potential hallucinations, we incorporate three technical innovations and one practical safeguard: 1) A human-simulated knowledge perception mechanism enforces step-by-step reasoning through a structured process of “knowledge retrieval→analysis→exercise solving”, improving transparency and reducing single-step generation errors. 2) We embed factual knowledge from domain-specific knowledge graphs into prompts and apply a dual-matching strategy to select relevant high-quality demonstrations, guiding generation toward pedagogically accurate and context-aware outputs. 3) An exercise generative-adversarial mechanism introduces expert agents to evaluate and refine exercises along three dimensions (linguistic fluency, coverage of erroneous knowledge concepts, and the correctness of exercises) through adversarial feedback. 4) As shown in the application interface screenshots in Appendix C, the generated exercises are editable in the application interface, allowing teachers to manually review and adjust final outputs before posting to students.\n\nIn summary, ExeGen combines LLM strengths with structured control and teacher-in-the-loop validation to reduce hallucination risks while enabling practical deployment. Notably, ExeGen is designed as an AI-assisted tool to support teachers rather than replace them. In this way, teachers shift from exercise creators to reviewers, enabling human-in-the-loop validation that aligns generated content with instructional goals. We appreciate your feedback and will explore advanced hallucination mitigation strategies such as reinforcement learning with human feedback and prompt tuning in future work.\n\n[1] SocraticLM: Exploring Socratic Personalized Teaching with Large Language Models, NeurIPS 2024.\n\n[2] What Makes In-context Learning Effective for Mathematical Reasoning, ICML 2025.\n\n**Response to W2&Q2:**\n\nThank you for the constructive suggestions. To strengthen the comparison, we add three advanced LLM reasoning variants as additional baselines, including Tree-of-Thoughts (ToT), Self-Consistency (SC), and Best-of-N (BoN). Due to time and space constraints, we evaluate these methods on the single-choice exercise setting. As shown in the following table, these advanced baselines slightly outperform the original CoT baseline. And our proposed ExeGen framework consistently outperforms all baselines across all metrics. These results further confirm the robustness and effectiveness of ExeGen.\n|Methods|KR|Clarity|AA|DA|EF|SE|HR|Recall|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|ToT|4.75|4.70|4.82|4.75|4.68|4.60|0.65|0.70|\n|SC|4.55|4.50|4.60|4.55|4.45|4.50|0.58|0.62|\n|BoN|4.60|4.55|4.65|4.60|4.50|4.55|0.60|0.65|\n|ExeGen|4.81|4.78|4.89|4.81|4.72|4.62|0.70|0.75|\n\nWe appreciate your suggestions, which help us further validate ExeGen’s superiority over recent LLM reasoning techniques.\n\n**Response to W3&Q3:**\n\nThank you for your insightful comments. We fully agree that subjective and open-ended exercises play a significant role in higher-order learning assessment. While the current paper focuses on objective exercise types with clear structure and evaluation criteria to ensure experimental rigor and reproducibility, we emphasize that the proposed ExeGen framework is inherently adaptable to generating subjective and open-ended exercises, such as code reading and completion.\n\nWe chose not to include these exercise types in the current paper for the following reasons: 1) Such exercises often lack unique answers and objective evaluation metrics, making automated evaluation difficult and less reliable; 2) Human evaluation for subjective or open-ended exercises requires domain expertise and is time-consuming, which limits scalability; 3) These exercises involve more complex text generation, increasing the risk of hallucination and higher token consumption. For these reasons, we focus on the objective exercise generation task that is more controllable and has quantifiable evaluation criteria in this paper.\n\nWe take your comments very seriously and are actively exploring the design of scalable human-computer collaborative evaluation mechanisms for subjective and open-ended exercise generation tasks. We would greatly appreciate the opportunity to further discuss the generation and assessment mechanisms for such exercises with you. Your professional expertise will provide valuable insights for our research. Once again, thank you for your attention and support."}]

Author rebuttal:

Meta-review: This paper introduces ExeGen, a novel multi-agent cooperation framework coordinates four specialized agents. The task and framework are clearly motivated, technically sound, and evaluated. As for the concern about evaluation based on GPT/humans and application in real classroom, I suggest the authors to give more explanations and discussions in the revised version.
Decision: Accept (poster)