Title: Knowledge Starts with Practice: Knowledge-Aware Exercise Generative Recommendation with Adaptive Multi-Agent Cooperation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Exercise Generative Recommendation

Knowledge State Perception

Personalized Learning

Multi-agent Cooperation
Abstract: Adaptive learning, which requires the in-depth understanding of students' learning processes and rational planning of learning resources, plays a crucial role in intelligent education. However, how to effectively model these two processes and seamlessly integrate them poses significant implementation challenges for adaptive learning. As core learning resources, exercises have the potential to diagnose students' knowledge states during the learning processes and provide personalized learning recommendations to strengthen students' knowledge, thereby serving as a bridge to boost student-oriented adaptive learning. Therefore, we introduce a novel task called Knowledge-aware Exercise Generative Recommendation (KEGR). It aims to dynamically infer students' knowledge states from their past exercise responses and customizably generate new exercises. To achieve KEGR, we propose an adaptive multi-agent cooperation framework, called ExeGen, inspired by the excellent reasoning and generative capabilities of LLM-based AI agents. Specifically, ExeGen coordinates four specialized agents for supervision, knowledge state perception, exercise generation, and quality refinement through an adaptive loop workflow pipeline. More importantly, we devise two enhancement mechanisms in ExeGen: 1) A human-simulated knowledge perception mechanism mimics students' cognitive processes and generates interpretable knowledge state descriptions via demonstration-based In-Context Learning (ICL). In this mechanism, a dual-matching strategy is further designed to retrieve highly relevant demonstrations for reliable ICL reasoning. 2) An exercise generation-adversarial mechanism collaboratively refines exercise generation leveraging a group of quality evaluation expert agents via iterative adversarial feedback. Finally, a comprehensive evaluation protocol is carefully designed to assess ExeGen. Extensive experiments on real-world educational datasets and a practical deployment in college education demonstrate the effectiveness and superiority of ExeGen. The code is available at https://github.com/dsz532/exeGen.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents ExeGen, a novel multi-agent framework for Knowledge-aware Exercise Generative Recommendation (KEGR). By coordinating LLM-powered agents for knowledge perception, exercise generation, and quality refinement, ExeGen provides personalized and interpretable learning support. Extensive experiments and real-world deployment validate its effectiveness in adaptive education.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed KEGR framework dynamically infers students’ knowledge states from their past exercise responses and customizes the generation of new exercises, making the design more aligned with real-world learning scenarios.\n2. The incorporation of a human-simulated knowledge perception mechanism and an adversarial refinement loop ensures both the interpretability of the learner's knowledge state and the high quality of the generated exercises.\n3. The proposed method has been deployed in real school settings, with application screenshots provided, which strongly demonstrates the practical validity of the approach.\n\nWeaknesses:\n1. To the best of my knowledge, even the most powerful large language models still suffer from hallucination issues. Since all the exercises in the paper are generated by LLMs, the proposed evaluation expert system—though carefully designed—cannot fundamentally guarantee that the generated exercises fully comply with educational standards.\n2. Due to the lack of directly comparable baselines, the authors adopt four representative methods. It may be worth considering the inclusion of some advanced LLM reasoning variants (e.g., Tree-of-Thoughts) to further demonstrate the effectiveness of the proposed approach.\n3. The framework currently supports only objective exercise types (single choice, multiple choice, judgment), omitting more complex subjective or open-ended tasks that are critical in higher-order learning assessment.\n\nQUESTIONS:\n1. In this work, all exercises are generated by LLMs. Although a corresponding evaluation expert system is designed, it still cannot fundamentally address the issue of ensuring that the generated exercises fully comply with educational standards, especially for complex problems.\n2. In addition to methods like CoT, I believe it would be beneficial to incorporate approaches such as Self-Consistency [1] and Best-of-N [2] to further validate the superiority of the proposed method.\n3. The current framework only supports objective exercise types (single choice, multiple choice, and judgment), and has not yet addressed other types of exercise.\n\n[1]Self-consistency improves chain of thought reasoning in language models. ICLR 2023.\n\n[2]Let’s verify step by step. ICLR 2024.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel and practical task _Knowledge-aware Exercise Generative Recommendation_, and introduces a LLM-based multi-agent framework. Its thoughtful design and real-world validation demonstrate strong potential for adaptive learning. The real-world deployment supports the claims of effectiveness and superiority.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. Bridges knowledge tracing and exercise recommendation, forming a closed-loop adaptive system.\n2. Four LLM agents (Manager, Perceiver, Generator, Evaluator) work in a feedback loop, effectively handling the KEGR task.\n3. The real-world deployment demonstrates its effectiveness.\n\nWeakness:\n1. While the proposed human-simulated knowledge perception mechanism aims to reduce LLM hallucination through a three-step process (knowledge retrieval, analysis, exercise solving), its effectiveness has not been independently validated.\n2. The current experiments focus on a single subject domain, but this is not clearly stated. Since knowledge structures differ significantly between disciplines, it remains unclear how well the proposed method generalizes across domains or whether subject-specific adjustments are needed.\n3. The domain knowledge graph used in this study lacks completeness, particularly in prerequisite relationships between concepts. This limitation may affect the accuracy of knowledge state modeling and the quality of generated exercises. A more detailed analysis of the knowledge graph’s coverage and its impact on system performance is needed.\n\nQUESTIONS:\n1. The manuscript acknowledges that LLMs may hallucinate. To address this, it introduces a human-simulated knowledge perception mechanism based on a domain knowledge graph, involving three steps: relevant knowledge retrieval, analysis, exercise solving. It is recommended to evaluate this pipeline experimentally and identify subject areas where it may not perform well.\n2. The experimental setup lacks clarity, especially regarding the subject domains (e.g., math, science). Since knowledge structures vary across subjects, it's important to test the method’s effectiveness in different domains and discuss whether the pipeline needs subject-specific adaptations.\n3. The completeness of the knowledge graph is crucial for accurate knowledge state estimation. A more thorough analysis of its coverage (especially missing prerequisite links), is needed to assess its impact on system performance.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Knowledge-aware Exercise Generative Recommendation (KEGR), a task that unifies knowledge state diagnosis and personalized exercise generation. To achieve this, the paper develops ExeGen, a multi-agent framework powered by LLMs, where four specialized agents collaborate in a closed-loop pipeline: Recommendation Manager, Knowledge Perceiver, Exercise Generator, and Quality Evaluation Experts. Furthermore, key techniques also include a human-like knowledge perception mechanism using knowledge graphs and in-context learning, and an adversarial refinement process to improve exercise quality. Experiments on real-world datasets and deployment in college settings show ExeGen’s superiority over existing baselines, validating its potential for adaptive and interpretable learning support.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper introduces KEGR, a meaningful task that integrates knowledge tracing and exercise recommendation into a unified, closed-loop adaptive learning system.\n\n2. ExeGen's use of four specialized LLM-powered agents—each responsible for a distinct subtask—ensures transparency and modularity in handling complex reasoning and generation tasks. Furthermore, Human-Simulated Knowledge Perception and Adversarial Quality Refinement strengthen the above agent’s ability, contributing to a more robust workflows.\n\n3. The system is evaluated comprehensively through a hybrid approach that includes GPT-based metrics, statistical scores, student questionnaires, and teacher expert assessments, which is not only tested offline but also deployed in a real college environment, providing both quantitative and qualitative validation and also supporting its practical utility and readiness for real-world application.\n\nWeakness:\n\n1. The proposed agent-based framework lacks substantial technical novelty. It heavily relies on existing techniques such as In-Context Learning (ICL) and Chain-of-Thought (CoT), and does not demonstrate significant innovation in the design of the agent architecture itself.\n\n2. The current evaluation focuses primarily on the quality of the generated exercises, assessed mainly through GPT-based scoring and human judgments. However, these metrics are inherently subjective and do not reflect the core objective of exercise recommendation—namely, the actual improvement in students' knowledge and learning outcomes as measured by their performance after doing exercised generated by this framework.\n\n3. While the motivation of the agent design is to address limitations in both knowledge tracing and exercise recommendation, the evaluation fails to provide a detailed comparison against traditional models from these domains. For instance, the Knowledge Perceiver module is intended to trace student knowledge states, yet its performance is not compared against standard knowledge tracing models, making it difficult to assess its effectiveness.\n\n4. Although the paper is the first to propose the KEGR task and solve it via an agent-based approach, there are already many well-established agent frameworks such as ReAct, Reflexion, etc. Including empirical comparisons with these frameworks, or clearly explaining why they are not suitable for KEGR, would significantly strengthen the contribution.\n\n5. The paper’s presentation is somewhat confusing. For example, in addition to the four core agent’s components, the authors introduce two mechanisms—Human-simulated Knowledge Perception and Exercise Generative-Adversarial mechanisms—but do not clearly specify how these techniques are integrated into which component of the framework. Clarifying their roles and placement within the overall framework would enhance clarity.\n\nQUESTIONS:\n1. Could the paper clarify what specific architectural or procedural innovations distinguish their multi-agent framework from simply coordinating ICL and CoT within standard LLM pipelines? What novel agent-specific interactions or functionalities are introduced beyond existing prompting paradigms?\n\n2. Have the paper conducted or considered any longitudinal studies to measure actual student learning gains after engaging with the generated exercises? If not, how do they justify the use of GPT-based and human scoring as sufficient proxies for real educational effectiveness?\n\n3. Given that the Knowledge Perceiver is designed to model student knowledge states, why is there no quantitative comparison with established knowledge tracing models such as BKT, DKT, or GKT? \n\n4. Many existing frameworks like ReAct and Reflexion have demonstrated strong performance in agent-based reasoning tasks. Could the authors provide empirical comparisons with these frameworks or explain in detail why such approaches are unsuitable for the KEGR task?\n\n5. How exactly are the Human-simulated Knowledge Perception Mechanism and the Exercise Generative-Adversarial Mechanism integrated into the four-agent architecture? Which agent(s) are responsible for executing these mechanisms, and how are they invoked in the workflow?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This work focuses on adaptive learning and proposes a novel task called Knowledge-aware Exercise Generative Recommendation (KEGR). To achieve good performance, the authors introduce an adaptive multi-agent cooperation framework, ExeGen, and explore techniques such as prompting and adversarial refinement. Experiments demonstrate the effects of the proposed method.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The authors effectively explain the motivation for this work in the introduction and clearly define the KEGR problem, including Knowledge Tracing (KT) and Exercise Recommendation (ER).

2. The concept of building a Human-simulated Knowledge Perception Mechanism is interesting.

Weaknesses:

1. The proposed framework heavily relies on data/analysis generated by GPT, which may introduce hallucination issues, even with techniques like prompting and adversarial refinement.

2. Most evaluation metrics are based on questionnaires filled out by GPT/humans, not convincing.

3. The correctness of generated knowledge state descriptions is crucial for high-quality exercise generation, but no evaluations on this aspect are provided.

4. The dataset's diversity is limited to only one Chinese dataset. More datasets are needed to demonstrate the framework’s generalization ability.

QUESTIONS:
Questions:

1. Is the Human-simulated Knowledge Perception Mechanism reasonable, given that different students have different knowledge bases?

2. Can this framework handle a large amount of historical response data?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03