From Objectives to Questions: A Planning-based Framework for Educational Mathematical Question Generation
Authors: Cheng Cheng, Zhenya Huang, GuanHao Zhao, Yuxiang Guo, Xin Lin, Jinze Wu, Xin Li, Shijin Wang
Accepted at: acl25

From Objectives to Questions: A Planning-based Framework for Educational Mathematical Question Generation

Cheng Cheng1,2, Zhenya Huang1,2,3*, Guanhao Zhao1,2, Yuxiang Guo1,2, Xin Lin1,2, Jinze Wu2,4, Xin Li2,4, Shijin Wang2,4
1 University of Science and Technology of China
2 State Key Laboratory of Cognitive Intelligence
3 Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
4 iFLYTEK Research
{doublecheng, ghzhao0223, guoyx18, linx}@mail.ustc.edu.cn
{huangzhy, leexin}@ustc.edu.cn
{jzwu4, sjwang3}@iflytek.com

Abstract
Automatically generating high-quality mathematical problems that align with educational objectives is a crucial task in NLP-based educational technology. Traditional generation methods focus primarily on textual quality, but they often overlook educational objectives. Moreover, these methods address only single-dimensional, simple question generation, failing to meet complex, multifaceted educational requirements. To address these challenges, we constructed and annotated EduMath, a dataset of 16k mathematical questions with multi-dimensional educational objectives. Based on this dataset, we developed EQGEVAL, which incorporates three evaluation dimensions and is designed to assess the ability of models to generate educational questions. Drawing inspiration from teachers’ problem design processes, we propose the Educational Question Planning with self-Reflection (EQPR) method for educational mathematical question generation, following a "plan-evaluate-optimize" approach. Specifically, by combining planning algorithm based on Monte Carlo Tree Search with the generative capabilities of Large Language Models, we continuously optimize questions through iterative feedback. This self-optimization mechanism ensures that the generated questions both fit the educational context and strategically achieve specific basic educational objectives. Through extensive experiments based on EQGEVAL, we have demonstrated that EQPR achieves significant improvements in generating questions that meet multi-dimensional educational objectives.1

1 Introduction
Mathematical questions are fundamental elements of educational assessment and cognitive development, playing an irreplaceable role in cultivating students’ logical thinking and problem-solving abilities (Kurdi et al., 2020; Liu et al., 2019). When designing such questions, educators coordinate multiple educational objectives(e.g., concepts, mathematical qualities ), not only to accurately evaluate students’ degree of knowledge mastery and application skills but also to enhance their comprehensive problem-solving capabilities through progressive challenges. Figure 1 illustrates the systematic process of how teachers design test questions (Kliebard, 1970; Wiggins, 2005). Starting with educational objectives, teachers carefully consider multiple dimensions including core concepts, competencies to be assessed, and real-world scenarios. They strategically combine these elements to create engaging questions that connect theoretical knowledge with practical applications. Throughout this process, teachers continuously evaluate whether their questions adequately cover all intended objectives and make necessary refinements.

However, in the literature, the generation of mathematics questions that consider multiple educational objectives has not received sufficient attention. In widely studied mathematical question generation, previous work has relied solely on manually crafted templates for generation (Polozov et al.,

![A simple example of question design based on educational objectives](page_1012_370_377_246.png)

Figure 1: A simple example of question design based on educational objectives
2015; Khodeir et al., 2018), unable to generate content based on educational objectives expressed in natural language. Conversely, most more advanced works employ seq2seq models for generation (Zhou and Huang, 2019; Liu et al., 2020), but due to model limitations, their applications remain confined to question generation based on mathematical formulas such as arithmetic operations and equation systems. With the emergence of large language models like ChatGPT (Kojima et al., 2022) and LLaMA (Dubey et al., 2024), which significantly enhance the ability to generate diverse and complex content, researchers have begun using few-shot learning or Chain-of-Thought templates for question generation, framing it as a goal-based reasoning task. However, question generation is a complex task, and single-step reasoning may lead to failures, as seen in the teacher’s first attempt in Figure 1. Consequently, the limitations of single-step reasoning—its inherent inability to handle complex problems requiring multi-step inferences and its lack of reflective capabilities to check for errors—significantly hinder the full potential of large language models in question generation.

To validate the use of large language models for creating educational questions, we empirically assess whether these models can match human teachers’ ability to design problems that achieve specific educational objectives. However, existing mathematics question generation datasets present significant limitations. Most datasets focus primarily on elementary-level mathematics (e.g., LMWP (Liu et al., 2020), HMWP (Qin et al., 2020)), offering a narrow scope of assessment. Furthermore, a lack of comprehensive annotation is prevalent; the majority of these datasets (e.g., Gaokao-bench (Zhang et al., 2023), GSM8K (Cobbe et al., 2021)) lack annotations for educational objectives, hindering a thorough evaluation of LLMs’ capabilities in mathematical question generation.

In this paper, to advance the field further, we present two datasets, EduMath-SQ and EduMath-CQ, which are filtered from real high school test papers and annotated with educational objectives. Specifically, for each question, we annotate four to five categories of educational objectives based on Tyler’s rationale (Kliebard, 1970) and two-way specification table (Odiagbe, 2016), aiming to comprehensively evaluate models’ objective-based mathematical question generation capabilities.

Along this direction, several technical challenges remain. First, there is a lack of evaluation metrics. Past question generation work has adopted text quality metrics such as BLEU and ROUGE. For instance, in Figure 1, while the initial and modified questions show high textual similarity, the initial question lacks a concept compared to the latter. However, these traditional text generation evaluation metrics often fail to effectively measure mathematical question generation quality, particularly in terms of educational objective alignment. Second, questions with multidimensional educational objectives typically cannot be generated in a single attempt, as the complexity of multiple objectives often leads to certain objectives being overlooked or missed in a single generation attempt, requiring repeatedly evaluated and optimized by educators. Additionally, a single dimension of an educational objective may encompass multiple components—as illustrated in Figure 1, where the "concepts" dimension includes both sine functions and elevation angles—further increasing the challenge of accurately generating questions that satisfy multidimensional educational objectives.

To address these challenges, we first propose EQGEVAL, a novel evaluation benchmark based on the EduMath dataset that comprehensively evaluates question generation quality through LLM using three metrics: solvability (question feasibility), Pass Rate (objective fulfillment), and Win Rate (comparison with gold standards). Furthermore, in real-world educational settings, teachers typically design mathematical questions through an iterative process, repeatedly evaluating and optimizing questions based on educational objectives and student feedback. Inspired by this practice, we develop a methodology that mirrors this systematic refinement process. Specifically, we develop the Educational Question Planning with self-Reflection (EQPR) method, which combines Monte Carlo Tree Search (MCTS) with large language models to simulate diverse question design strategies, achieve educational objectives, and systematically evaluate and refine the question creation process through continuous feedback.

In summary, our contributions are:

• We introduce a comprehensive dataset derived from authentic educational assessments, together with EQGEVAL, a novel benchmark framework designed to evaluate large language models’ proficiency in generating questions that are precisely aligned with specified educational objectives.
• Our framework, EQPR, integrates Monte Carlo Tree Search (MCTS) for strategic planning with a reflection mechanism that evaluates question quality and proposes improvements, enabling iterative refinement of the generated questions.

• Empirical validation through extensive experimentation underscores our framework’s efficacy. When implemented with two distinct LLMs and evaluated on the EQGEVAL benchmark, EQPR achieved superior performance across nearly all evaluation metrics, establishing new state-of-the-art results.

2 Related Works

2.1 Question Generation

Question generation is a significant research direction in educational technology (Kurdi et al., 2020; Zhao et al., 2024a,b). Its core function lies in automatically generating questions from structured or natural language text, such as deriving questions from dialogue content (Guo et al., 2024; Liu et al., 2024) or extracting them from story texts (Li and Zhang, 2024). The key value of this technology is that it can substantially reduce the time and cost of manual design and construction of questions, while also dynamically generating questions of varying difficulty and type based on the content.

In mathematics education, question generation has been a significant area of research. Early work focused on generating mathematical questions based on mathematical formulas and scenarios, relying on predefined templates and rule-based methods. Researchers developed language knowledge bases and rhetorical structure rules to aid in question generation (Khodeir et al., 2018). As natural language processing evolved, the field shifted toward neural network-based approaches. Sequence-to-sequence frameworks for mathematical question generation (Zhou and Huang, 2019) integrated thematic and formulaic information using attention mechanisms. Building on pre-trained language models, later work improved topic word selection and introduced question-solving modules to enhance the solvability of generated questions (Wang et al., 2021). MapKG (Qin et al., 2023a), inspired by educational experts’ test design experience, advanced the field with a "plan-then-generate" strategy that incorporated dual attention mechanisms and knowledge graphs to improve question solvability and diversity. Recent developments in large language models have opened new research avenues, with methods such as gradient-based techniques being used to generate questions with controlled difficulty levels (Lin et al., 2024). However, existing studies have largely overlooked the generation of questions that address multi-dimensional educational objectives. Our proposed EQPR model aims to fill this gap.

2.2 LLM Planning and Reflection

Recent research on planning with large language models (LLMs) has witnessed remarkable advancement, significantly impacting domains such as common-sense reasoning (Zhu et al., 2023; Xue et al., 2024) and embodied intelligence (Sun et al., 2024). A central focus has been enhancing LLMs’ capacity for systematic, step-by-step problem solving. Foundational techniques like Chain-of-Thought (CoT) prompting (Wei et al., 2022) have established the paradigm of incremental problem decomposition, while more sophisticated approaches such as Tree-of-Thoughts (ToT)(Yao et al., 2024) and Reasoning via Planning (RAP)(Hao et al., 2023) explore solution spaces through hierarchical tree structures, leveraging methodologies like Monte Carlo Tree Search to systematically expand the search space. However, extensive iterative planning processes can introduce cumulative error propagation. To address this limitation, Self-Refine (Madaan et al., 2024) incorporates iterative feedback mechanisms that enable models to reflect upon and refine their reasoning processes, thereby mitigating error accumulation. Distinguished from these planning-focused methodologies, our work introduces EQPR, which synergistically combines iterative planning with reflective mechanisms to optimize question generation processes.

3 Problem Formulation

As outlined in Section 1, our objective is to leverage large language models for generating questions that align with specified educational objectives \( O \), encompassing fundamental goals (e.g., conceptual understanding, Bloom’s taxonomy levels) and their contextual frameworks (e.g., traditional cultural narratives, sports scenarios), formalized as \( q = LLM(O) \).

However, single-pass question generation often yields limited quality and inadequate alignment
Figure 2: The overall framework of EQPR. The upper part of the figure illustrates the entire EQPR process, which uses an MCTS structure for deep reasoning and iteratively improves question quality via the Critic and Reflection modules. The lower part shows a simplified state transition example, where a previous question is modified based on feedback and progresses to the next state.

with educational objectives. To address this limitation, we draw inspiration from teachers’ iterative improvement processes, modeling this as a sequential decision-making problem and formalizing it as a Markov Decision Process (MDP).

We formalize this question improvement process as an MDP defined by the tuple \((\mathcal{S}, \mathcal{A}, \mathcal{T}, R)\), where:

• State space \( \mathcal{S} \): Each state \( s_t \in \mathcal{S} \) represents the current version of an educational question at time step \( t \), denoted as \( q_t \). States may include metadata such as alignment with educational objectives \( O \), revision history, or evaluative metrics.

• Action space \( \mathcal{A} \): Each action \( a_t \in \mathcal{A} \) corresponds to an improvement operation targeting specific dimensions of the question, such as “enhancing conceptual clarity” or “increasing real-world relevance.” As shown in the upper part of Figure 2, these actions are formulated as structured feedback that guides iterative improvement, offering detailed analysis of current shortcomings paired with actionable improvement strategies.

• Reward function \( R : \mathcal{S} \times \mathcal{A} \to \mathbb{R} \): After executing action \( a_t \) in state \( s_t \), the system receives a reward based on the revised question’s quality. This reward reflects multiple educational criteria including conceptual clarity, cognitive depth, and contextual appropriateness, assessed by a critic module providing both numerical scores and formative feedback (detailed in Section 4.2).

• Transition function \( \mathcal{T} : \mathcal{S} \times \mathcal{A} \to \mathcal{S} \): The system transitions to a new state \( s_{t+1} \) after applying action \( a_t \) to the current question \( s_t \). This process is implemented by a reflection module, which interprets the critic’s feedback and generates an improved question accordingly (detailed in Section 4.3).

As illustrated in Figure 2, the process operates as follows: Given a current state \( s_t \) and educational objectives \( O \), the critic module analyzes the question and samples an improvement action \( a_t \sim \text{Critic}(a \mid s_t, O) \). Simultaneously, it assigns a reward \( r_t \) and identifies specific improvement areas. This feedback is passed to the reflection module, which translates it into actionable revision instructions and generates an enhanced question better aligned with educational objectives. For instance, if the action is to “increase cognitive depth,” the reflection module may introduce more abstract or higher-order reasoning components. The newly
generated question becomes the next state \( s_{t+1} \), and the process continues iteratively until optimal educational alignment is achieved.

4 Method

4.1 Overview
Drawing inspiration from educators’ systematic approach to question development (Wang et al., 2022) - which encompasses planning, writing, evaluating, and optimizing - we introduce an innovative framework for generating educational questions that ensures both quality and alignment with educational objectives. The framework consists of three modules: the Critic module, which evaluates each generated question (state) based on multiple dimensions of educational objectives and provides directions for modification; the Reflection module, which analyzes the feedback from the Critic to determine optimization directions and refine the question generation process, leading to the creation of new questions; and the MCTS-based Planning module, directed by the Critic and Reflection modules, which provide the necessary guidance and constraints to enable it to systematically and efficiently navigate the vast and multifaceted search space, thereby exploring a wide range of potential question structures and formulations.

4.2 Critic
We employ the Critic module to systematically evaluate the alignment between generated questions and educational objectives. Through Large Language Models (LLMs) equipped with Critic prompts (see Table 8 ) and Chain-of-Thought reasoning, the module performs comprehensive assessment across multiple dimensions. Specifically, for a state \( s_t \) at phase t and educational objective \( O \), the Critic generates question modification directions \( a_t \) and \( score_t \) as follows:

\[
(score_t, a_t) = Critic(s_t, O),
\]

Here, \( score_t \) represents the quantitative evaluation score, while \( a_t \) denotes the suggested modification directions. The evaluation examines concept coverage, contextual relevance, conceptual coherence, etc., ensuring comprehensive assessment of educational requirements, and logical interconnections. This structured evaluation framework enables precise identification of gaps between generated questions and desired educational objectives, facilitating targeted improvements in subsequent iterations.

4.3 Reflection
Cognitive science research demonstrates that humans continuously iterate and reflect upon their thinking based on new information (Frederick, 2005), allowing them to correct errors and deepen understanding. This reflective process is particularly evident in educational settings, where teachers systematically refine questions through iterative evaluation and improvement to align with educational objectives.

Consider the example illustrated in the bottom part of Figure 2, where the initial question generation lacks contextual materials. Through reflection, this limitation is identified, leading to an improved iteration that incorporates a practical tree-planting scenario, making the mathematical concept more accessible and applicable.

To formalize this reflective iteration process, we introduce a Reflection module powered by large language models. This module employs specialized reflection prompts (detailed in the Table 9) to analyze the historical trajectory of question states and improvement suggestions, ensuring alignment with educational objectives. In the iteration step \( t \), we construct the historical trajectory \( \tau \) as:

\[
\tau = \{(s_0, a_0), (s_1, a_1), \ldots, (s_{t-1}, a_{t-1})\},
\]

where \( s_0 \) represents the initial question state generated based on the initial generation prompt (see Table 7) at the root node, \( a_0 \) denotes the Critic module’s modification suggestions for initial question \( s_0 \), \( s_t \) indicates the preceding state of the current node, and \( a_t \) represents the Critic module’s modification suggestions for question \( s_t \). This trajectory captures the complete history of question states and their corresponding improvement suggestions up to the current iteration. The next state generation is accomplished through the Reflection module:

\[
s_{t+1} = Reflection(O, \tau, s_t, a_t),
\]

where \( O \) represents the educational objectives.

This formalized reflection mechanism enables the question generation process to emulate human teachers’ approach, continuously iterating to enhance problem quality.

4.4 MCTS-based Planning
We combine large language models (LLMs) with Monte Carlo Tree Search (MCTS) to better meet
our educational objectives. MCTS efficiently explores the question-generation space, balancing exploration and exploitation to improve the educational value of the resulting questions. As shown in the top panel of Figure 2, we model the process as a search tree: each node represents a partially generated question, and each edge denotes an editing action—such as extending or rewriting the text. The overall planning loop follows the standard MCTS phases of selection, expansion, simulation, and backpropagation, detailed in Algorithm 1 in the Appendix.

Selection. The selection phase chooses the most promising node from the tree’s branches for further exploration. Starting from the root node \( s_0 \) (initial question), the selection phase iteratively chooses the most promising nodes for exploration. To balance between known high-quality question structures (exploitation) and exploring new directions for improvement (exploration), we use the well-known Upper Confidence Bounds applied to Trees (UCT) (Kocsis and Szepesvári, 2006) for node selection, as shown below:

\[
UCT(s_t, a_t) = Q(s_t, a_t) + c \sqrt{\frac{\ln N(s_t)}{N(ch(s_t, a_t))}},
\]

where \( Q(s_t, a_t) \) is the potential future reward of applying action \( a_t \) at time \( t \), \( N(s_t) \) is the number of visits to node \( s_t \), \( ch(s_t, a_t) \) is the child node reached after executing action \( a_t \) in state \( s_t \), and \( c \) is a constant used to adjust exploration. At each level of the tree, the child node with the highest UCT value is selected.

Expansion. During the expansion phase, we utilize the Reflection module to generate new question states. This module leverages the historical optimization trajectory \( \tau \) to analyze patterns from previous modifications, generating new candidate questions according to: \( s_{t+1} = Reflection(O, \tau, s_t, a_t) \). To explore a wider range of question designs, multiple nodes are generated as child nodes in each expansion step.

Simulation. From the expanded nodes, the simulation phase explores potential question optimization paths via simulations. In each simulation, the system evaluates options based on estimated cumulative rewards, selecting the highest-reward path for further exploration. This continues until a terminal state, yielding a comprehensive simulation of potentially effective question optimization schemes.

Backpropagation. When a simulation reaches a terminal state, backpropagation begins, using the cumulative future reward obtained at the terminal node (illustrated by the red arrows in the central MCTS tree section of Figure 2) to update the Q-values of each state-action pair, with the aim of refining future question selection.

5 Educational QG Dataset & Benchmark

Existing mathematical question generation datasets primarily focus on elementary-level content and often lack comprehensive educational annotations. To address these limitations, we developed EduMath, aiming to create a high-quality dataset with annotation accuracy exceeding 90% across all educational dimensions. We developed this multi-dimensional annotation framework based on the following considerations: concept mapping reflects knowledge coverage, ability assessment captures problem-solving requirements, Bloom’s Taxonomy represents cognitive levels, mathematical literacy evaluation aligns with educational objectives, and real-world context identification demonstrates practical value. These dimensions characterize the educational attributes of mathematical questions from distinct perspectives. We sourced 16k high-school level mathematics problems from mock exams and college entrance examinations, implementing strict quality filters to exclude problems with images or incomplete solutions. Using DeepSeek-V3, we conduct an iterative three-round annotation process. Initial annotations are reviewed for accuracy and consistency. Specifically, we employ Chain-of-Thought prompting to guide multiple large language models in evaluating annotation correctness through a voting mechanism. Annotations flagged as inaccurate are re-annotated to resolve identified issues. This rigorous, multi-stage process results in an annotation accuracy of 95.2% across all dimensions. The final dataset includes two variants: EduMath-SQ (Standard Questions) and EduMath-CQ (Contextual Questions), with EduMath-SQ focusing exclusively on non-contextual problems.

Building on this, we define controllable educational question generation and corresponding evaluation metrics. Previous question generation approaches typically rely on text quality metrics such as BLEU and ROUGE for evaluation, but these are far from sufficient. Truly effective questions must be solvable and meet teachers’ instructional and assessment needs. Following TOOLEVAL (Qin et al., 2023b) and based on DeepSeek-V3, we propose EQGEVAL, which includes the following metrics
(see Appendix A.4 for details): Solvability as a fundamental requirement that the generated problems must have valid solutions; Pass Rate measuring the proportion of generated problems that meet the educational objectives; and Win Rate where we present educational objectives and two problems to DeepSeek-V3 evaluators, asking them to determine which problem better serves the intended purpose. To ensure the reliability of our evaluation, all metrics are determined through a majority voting mechanism, where multiple independent evaluations are conducted to derive the final assessment results.

6 Experiment

6.1 Experimental Settings

Datasets. As outlined in Section 5, we conducted experimental evaluations using the EduMath-SQ and EduMath-CQ datasets. We randomly selected 10% of the data to serve as the test set. The EduMath-CQ dataset comprises 589 educational objectives paired with their corresponding gold-standard questions, while EduMath-SQ contains 1,034 educational objectives along with their respective gold-standard questions. For more detailed information about these datasets, please refer to Appendix A.1.

Baselines. Since this is a novel task, we approach it as a reasoning problem and benchmark it against established reasoning methods, including Chain-of-Thought (CoT) (Wei et al., 2022), CoT-BON, ReAct (Yao et al., 2023), and the tree-structured DEAR method (Xue et al., 2024). DeepSeek-V3 (Deepseek, 2024) and GPT-4o-Mini (OpenAI, 2024a) serve as our primary backbone models. Additionally, we evaluated Claude-3.5 (Anthropic, 2024) and GPT-4o (OpenAI, 2024b) on the EduMath-CQ dataset; for those results, please refer to Appendix B.

Metrics. We evaluate our approach using both automatic metrics and human assessment. For automatic evaluation, we primarily employ three metrics from the EQGEVAL framework, where the Win Rate metric compares generated questions against gold-standard problems. To comprehensively assess text quality, we also incorporate widely-adopted natural language generation metrics, including Rouge-L (Lin and Och, 2004), BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and BERTSCORE (Zhang et al., 2020). For human evaluation, we design two metrics: Fluency and Human-rated Win Rate. The Fluency metric uses a three-level scoring system (0-2), with detailed scoring criteria provided in the Appendix A.7.

6.2 Main Results

We conduct experiments to verify the effectiveness of our framework EQPR, and report the results in Table 1. We get the following observations. Primarily, our method demonstrates superior performance in Win Rate and Pass Rate across both EduMath-CQ and EduMath-SQ datasets, as well as across both GPT-4o-Mini and DeepSeek models. This comprehensive outperformance validates the effectiveness of our approach. Notably, when testing with DeepSeek on EduMath-CQ, our method achieved a Win Rate of 46.23%, surpassing the next-best method DEAR (41.8%) by a substantial margin of 4.42%. This improvement clearly demonstrates the efficacy of our iterative refinement strategy. Nevertheless, we observed that Pass Rates consistently decreased when transitioning from standard questions (EduMath-SQ) to contextual scenarios (EduMath-CQ), suggesting that large language models still face challenges in seamlessly incorporating contextual elements while addressing multiple educational objectives.

Regarding Solvability, our method achieves near-optimal performance, ranking second across both sub-datasets. The slight reduction in solvability scores can be attributed to the nature of our iterative refinement process, where questions naturally evolve to become more sophisticated through multiple iterations.

In terms of traditional text quality metrics, our method maintains competitive performance while prioritizing educational effectiveness. For instance, in the EduMath-SQ dataset with DeepSeek, the modest difference between our method’s BERTScore (74.79) and REACT’s (76.05) is acceptable, considering that these metrics primarily assess lexical and semantic similarities rather than educational value. Our superior Win Rate and Pass Rate scores underscore our method’s success in achieving its primary objective: generating educationally meaningful and high quality questions, even if this occasionally leads to slightly lower linguistic metric scores.

6.3 Human Evaluation Results

To thoroughly assess the effectiveness of our methodology, we carried out human evaluation
Table 1: Evaluation results on datasets EduMath-CQ and EduMath-SQ(%).

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Method</th>
    <th colspan="6">GPT-4o-Mini</th>
    <th rowspan="2">SOLVABLE</th>
    <th rowspan="2">PASS RATE</th>
  </tr>
  <tr>
    <th>BLEU</th>
    <th>METEOR</th>
    <th>ROUGE-L</th>
    <th>BERTSCORE</th>
    <th>WIN RATE</th>
  </tr>
  <tr>
    <td rowspan="5">EduMath-CQ</td>
    <td>COT</td>
    <td>11.60</td>
    <td>29.64</td>
    <td>24.45</td>
    <td>68.52</td>
    <td>35.78</td>
    <td>90.12</td>
    <td>27.88</td>
  </tr>
  <tr>
    <td>COT-BON</td>
    <td>8.52</td>
    <td>22.59</td>
    <td>10.56</td>
    <td>69.19</td>
    <td>34.50</td>
    <td>90.23</td>
    <td>21.25</td>
  </tr>
  <tr>
    <td>REACT</td>
    <td>18.90</td>
    <td><b>53.20</b></td>
    <td>36.72</td>
    <td><b>72.55</b></td>
    <td>38.40</td>
    <td>89.32</td>
    <td>33.66</td>
  </tr>
  <tr>
    <td>DEAR</td>
    <td>13.31</td>
    <td>23.09</td>
    <td>21.74</td>
    <td>70.34</td>
    <td>35.18</td>
    <td>85.77</td>
    <td>27.35</td>
  </tr>
  <tr>
    <td>EQPR</td>
    <td><b>24.80</b></td>
    <td>52.70</td>
    <td><b>48.13</b></td>
    <td>70.89</td>
    <td><b>39.92</b></td>
    <td><b>90.63</b></td>
    <td><b>35.81</b></td>
  </tr>
  <tr>
    <td rowspan="5">EduMath-SQ</td>
    <td>COT</td>
    <td>12.11</td>
    <td>42.90</td>
    <td>22.51</td>
    <td>70.64</td>
    <td>38.20</td>
    <td>90.84</td>
    <td>31.45</td>
  </tr>
  <tr>
    <td>COT-BON</td>
    <td>13.58</td>
    <td>43.60</td>
    <td>28.93</td>
    <td>70.85</td>
    <td>38.99</td>
    <td>90.25</td>
    <td>29.63</td>
  </tr>
  <tr>
    <td>REACT</td>
    <td><b>21.76</b></td>
    <td>47.98</td>
    <td><b>47.06</b></td>
    <td>70.32</td>
    <td>39.75</td>
    <td>90.64</td>
    <td>38.91</td>
  </tr>
  <tr>
    <td>DEAR</td>
    <td>20.71</td>
    <td><b>49.06</b></td>
    <td>42.11</td>
    <td><b>74.48</b></td>
    <td>41.81</td>
    <td><b>92.59</b></td>
    <td>37.88</td>
  </tr>
  <tr>
    <td>EQPR</td>
    <td>20.33</td>
    <td>46.86</td>
    <td>44.57</td>
    <td>71.02</td>
    <td><b>46.23</b></td>
    <td>91.73</td>
    <td><b>43.11</b></td>
  </tr>
  <tr>
    <td rowspan="5">EduMath-SQ</td>
    <td>COT</td>
    <td>1.76</td>
    <td>21.04</td>
    <td>7.64</td>
    <td>67.30</td>
    <td>32.20</td>
    <td>83.65</td>
    <td>64.60</td>
  </tr>
  <tr>
    <td>COT-BON</td>
    <td>2.63</td>
    <td>30.44</td>
    <td>11.76</td>
    <td>68.43</td>
    <td>36.64</td>
    <td><b>85.23</b></td>
    <td>75.12</td>
  </tr>
  <tr>
    <td>REACT</td>
    <td><b>36.72</b></td>
    <td>52.19</td>
    <td>36.72</td>
    <td><b>72.55</b></td>
    <td>36.81</td>
    <td>82.90</td>
    <td>83.22</td>
  </tr>
  <tr>
    <td>DEAR</td>
    <td>8.51</td>
    <td>20.52</td>
    <td>4.71</td>
    <td>66.34</td>
    <td>36.70</td>
    <td>81.89</td>
    <td>82.36</td>
  </tr>
  <tr>
    <td>EQPR</td>
    <td><b>32.77</b></td>
    <td><b>52.75</b></td>
    <td><b>52.78</b></td>
    <td>68.83</td>
    <td><b>37.18</b></td>
    <td>84.40</td>
    <td><b>84.37</b></td>
  </tr>
  <tr>
    <td rowspan="5">EduMath-SQ</td>
    <td>COT</td>
    <td>28.80</td>
    <td>56.91</td>
    <td>50.33</td>
    <td>75.38</td>
    <td>37.40</td>
    <td>87.15</td>
    <td>85.46</td>
  </tr>
  <tr>
    <td>COT-BON</td>
    <td><b>30.66</b></td>
    <td><b>61.50</b></td>
    <td><b>52.07</b></td>
    <td><b>76.05</b></td>
    <td>42.70</td>
    <td>86.88</td>
    <td>88.82</td>
  </tr>
  <tr>
    <td>REACT</td>
    <td>26.46</td>
    <td>60.47</td>
    <td>46.05</td>
    <td>75.45</td>
    <td>44.45</td>
    <td>85.95</td>
    <td>89.76</td>
  </tr>
  <tr>
    <td>DEAR</td>
    <td>24.52</td>
    <td>58.23</td>
    <td>40.07</td>
    <td>73.48</td>
    <td>40.36</td>
    <td>87.14</td>
    <td>82.74</td>
  </tr>
  <tr>
    <td>EQPR</td>
    <td>29.71</td>
    <td>57.29</td>
    <td>49.63</td>
    <td>74.79</td>
    <td><b>45.65</b></td>
    <td><b>91.31</b></td>
    <td><b>91.50</b></td>
  </tr>
</table>

<table>
  <tr>
    <th>Method</th>
    <th>Clarity</th>
    <th>Win Rate-Human(%)</th>
  </tr>
  <tr>
    <td>COT</td>
    <td>1.83</td>
    <td>27.00</td>
  </tr>
  <tr>
    <td>COT-BON</td>
    <td>1.78</td>
    <td>33.67</td>
  </tr>
  <tr>
    <td>DEAR</td>
    <td>1.84</td>
    <td>31.00</td>
  </tr>
  <tr>
    <td>REACT</td>
    <td>1.87</td>
    <td>35.00</td>
  </tr>
  <tr>
    <td>EQPR</td>
    <td><b>1.93</b></td>
    <td><b>36.67</b></td>
  </tr>
  <tr>
    <td>Fleiss’ kappa</td>
    <td>0.71</td>
    <td>0.47</td>
  </tr>
</table>

Table 2: Human evaluation results on EduMath-CQ(DeepSeek-V3).

experiments. We randomly selected 100 samples from the EduMath-CQ dataset and enlisted three highly educated evaluators, each with at least a bachelor’s degree, to conduct the assessments. The evaluation focused on two key dimensions: question clarity (whether the questions are easily readable and understandable) and quality comparison. Given the potential biases in large language models when determining win rates, the evaluators were asked to compare the quality of the generated questions against the gold-standard questions. As depicted in Table 2, our method demonstrated superior performance in both clarity scores and human-evaluated win rates. Furthermore, we evaluated the reliability of the annotations using Fleiss’ Kappa coefficient. The Kappa values for both clarity and quality comparison exceeded the credibility threshold of 0.41, thereby confirming the reliability of our evaluation results.

6.4 Ablation Results

Effect of Reflection Module. To validate the effectiveness of the Reflection Module, we conducted a comparative experiment. After removing the Reflection Module, the model no longer optimizes based on iterative feedback but directly generates questions according to the target (referred to as the "w/o Reflection" experiment). The results showed significant performance degradation across both models: using DeepSeek as an example, illustrated in Figure 3, the Pass Rate decreased from 46.23% to 43.51%, and the Win Rate declined from 43.11% to 40.74%. These observations highlight the vital importance of the Reflection Module. By engaging in multiple rounds of iterative optimization, the system adeptly integrates knowledge from previously encountered questions, enabling it to produce content that more effectively aligns with educational objectives.

Effect of MCTS-based Planning. To quantify the contribution of MCTS-based planning within our proposed framework, we conducted an ablation study by removing the MCTS component and employing greedy search exclusively for action selection. As demonstrated in Figure 3, the model’s per-
Figure 3: The results of ablation studies. We test different methods on EduMath-CQ datasets

formance deteriorated substantially without MCTS-based planning. Specifically, DeepSeek experienced a notable decline in both pass rate (from 46.23% to 41.57%) and win rate (from 43.11% to 38.79%) when operating without Monte Carlo Tree Search. We attribute this performance degradation to MCTS-based planning’s superior capability in navigating the question optimization space through its dual mechanism of prospective outcome prediction and retrospective evaluation. These empirical findings validate the critical role of MCTS integration in our framework’s effectiveness.

7 Conclusion

In this paper, we introduced EQPR (Educational Question Planning with self-Reflection), an innovative framework for generating high-quality mathematical questions that align with educational objectives. EQPR integrates a "plan-evaluate-optimize" process, combining Monte Carlo Tree Search with the generative power of LLMs, enabling continuous refinement through feedback optimization. We also introduced EduMath, a high-quality dataset of 16k mathematics problems, and EQGEVAL, a comprehensive framework for evaluating the educational value of generated questions. Extensive experiments demonstrate that EQPR outperforms existing reasoning methods on key educational metrics across multiple large language models.

Acknowledgement

This research was partially supported by the National Science and Technology Major Project(No.2022ZD0117103), the National Natural Science Foundation of China (Grants No.62477044), Anhui Provincial Natural Science Foundation (No. 2308085QF229), the Fundamental Research Funds for the Central Universities (No.WK2150110038). Zhenya Huang gratefully acknowledges the support of the Young Elite Scientists Sponsorship Program by CAST (No. 2024QNRC001)

Limitations

Our research primarily focused on mathematical question generation and has not yet been extended to other subject areas. This presents an important direction for future research. Additionally, we face certain challenges in establishing educational objectives, particularly in the assessment of question difficulty. Since difficulty evaluation is largely subjective and challenging to standardize, this remains a significant hurdle in the field that requires further investigation. Furthermore, while we utilize large language models for evaluation, these models may exhibit certain biases, and their assessment results do not always align perfectly with the professional judgment of human educators. Consequently, exploring effective methods to align human evaluation with LLM-based assessments and establishing a more accurate evaluation system remains a crucial direction for future research.

References

Anthropic. 2024. Introducing claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet.

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, pages 65–72.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Deepseek. 2024. DeepSeek-V3. https://huggingface.co/deepseek-ai/DeepSeek-V3.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Shane Frederick. 2005. Cognitive reflection and decision making. Journal of Economic perspectives, 19(4):25–42.

Shasha Guo, Lizi Liao, Jing Zhang, Cuiping Li, and Hong Chen. 2024. Pcqpr: Proactive conversational question planning with reflection. arXiv preprint arXiv:2410.01363.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992.

Nabilah Ahmed Khodeir, Hanan Elazhary, and Nayer Wanas. 2018. Generating story problems via controlled parameters in a web-based intelligent tutoring system. The International Journal of Information and Learning Technology, 35(3):199–216.

Herbert M Kliebard. 1970. The tyler rationale. The School Review, 78(2):259–272.

Levente Kocsis and Csaba Szepesvári. 2006. Bandit based monte-carlo planning. In European conference on machine learning, pages 282–293. Springer.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.

Ghader Kurdi, Jared Leo, Bijan Parsia, Uli Sattler, and Salam Al-Emari. 2020. A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education, 30:121–204.

Kunze Li and Yu Zhang. 2024. Planning first, question second: An llm-guided method for controllable question generation. In Findings of the Association for Computational Linguistics ACL 2024, pages 4715–4729.

Chin-Yew Lin and FJ Och. 2004. Looking for a few good metrics: Rouge and its evaluation. In Nicir workshop, pages 1–8.

Fan Lin, Shuyi Xie, Yong Dai, Wenlin Yao, Tianjiao Lang, Zishan Xu, Zhichao Hu, Xiao Xiao, Yuhong Liu, and Yu Zhang. 2024. Idgen: Item discrimination induced prompt generation for llm evaluation. arXiv preprint arXiv:2409.18892.

Jiayu Liu, Zhenya Huang, Tong Xiao, Jing Sha, Jinze Wu, Qi Liu, Shijin Wang, and Enhong Chen. 2024. Socraticlm: Exploring socratic personalized teaching with large language models. Advances in Neural Information Processing Systems, 37:85693–85721.

Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. 2019. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering, 33(1):100–115.

Tianqiao Liu, Qiang Fang, Wenbiao Ding, Hang Li, Zhongqin Wu, and Zitao Liu. 2020. Mathematical word problem generation from commonsense knowledge graph and equations. arXiv preprint arXiv:2010.06196.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36.

Sunny Ighalo Odiagbe. 2016. Table of specification: Ensuring content validity of teacher-made-test among senior secondary schools teachers in kwali area council. Journal of the Nigerian Academy of Education Vol, 13(1):69.

OpenAI. 2024a. GPT-4o mini: Advancing cost-efficient intelligence. https://openai.com/index/gpt-4o-mini.

OpenAI. 2024b. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, ACL 2002, pages 311–318.

Oleksandr Polozov, Eleanor O’Rourke, Adam M Smith, Luke Zettlemoyer, Sumit Gulwani, and Zoran Popovic. 2015. Personalized mathematical word problem generation. In Twenty-Fourth International Joint Conference on Artificial Intelligence.

Jinghui Qin, Lihui Lin, Xiaodan Liang, Rumin Zhang, and Liang Lin. 2020. Semantically-aligned universal tree-structured solver for math word problems. arXiv preprint arXiv:2010.06823.

Longhu Qin, Jiayu Liu, Zhenya Huang, Kai Zhang, Qi Liu, Binbin Jin, and Enhong Chen. 2023a. A mathematical word problem generator with structure planning and knowledge enhancement. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1750–1754.

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023b. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789.

Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. 2024. Adaplaner: Adaptive planning from feedback with language models. Advances in Neural Information Processing Systems, 36.

Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric P Xing, and Zhiting Hu. 2023. Promptagent: Strategic planning with language models enables expert-level prompt optimization. arXiv preprint arXiv:2310.16427.

Xu Wang, Simin Fan, Jessica Houghton, and Lu Wang. 2022. Towards process-oriented, modular, and versatile question generation that meets educational needs. arXiv preprint arXiv:2205.00355.
Zichao Wang, Andrew S Lan, and Richard G Baraniuk. 2021. Math word problem generation with mathematical consistency and problem context constraints. arXiv preprint arXiv:2109.04546.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Grant Wiggins. 2005. Understanding by design. Association for Supervision and Curriculum Development.

Shangzi Xue, Zhenya Huang, Jiayu Liu, Xin Lin, Yuting Ning, Binbin Jin, Xin Li, and Qi Liu. 2024. Decompose, analyze and rethink: Solving intricate problems with human-like reasoning cycle. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020.

Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. 2023. Evaluating the performance of large language models on gaokao benchmark. arXiv preprint arXiv:2305.12474.

Guanhao Zhao, Zhensha Huang, Yan Zhuang, Haoyang Bi, Yiyuan Wang, Fei Wang, Zhiyuan Ma, and Yixia Zhao. 2024a. A diffusion-based cognitive diagnosis framework for robust learner assessment. IEEE Transactions on Learning Technologies.

Hongke Zhao, Likang Wu, Yuqing Shan, Zonghan Jin, Yuanpei Sui, Zipeng Liu, Nan Feng, Minqiang Li, and Wei Zhang. 2024b. A comprehensive survey of large language models in management: Applications, challenges, and opportunities. Challenges, and Opportunities (August 14, 2024).

Qingyu Zhou and Danqing Huang. 2019. Towards generating math word problems from equations and topics. In Proceedings of the 12th international conference on natural language generation, pages 494–503.

Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2023. Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations.

A More Experiment Details

A.1 Dataset Details

As shown in the Table 4, we present the number of questions for each dataset along with the average number of concepts covered per question. On average, each question involves more than two concepts, indicating that we intend to use these datasets to evaluate the ability of large language models to generate questions that address complex educational objectives. To this end, we randomly sampled 10% of the questions from each dataset to serve as a test set.

A.2 Explanation of the Educational Objectives

• Concept: Understand and master core mathematical concepts such as trigonometric functions, sequences, and probability.

• Core Quality: Develop essential skills such as logical reasoning, mathematical modeling, and problem-solving to tackle complex mathematical tasks.

• Core Ability: Develop the ability to choose and apply appropriate mathematical techniques—for instance, recognizing when to use identities like \( \sin^2 x + \cos^2 x = 1 \) in solving problems.

• Bloom Level: Design questions that intentionally target specific cognitive levels in Bloom’s Taxonomy, such as application, analysis, or creation.

• Context: Enable students to interpret and solve mathematical problems within real-world scenarios and authentic contexts.

A.3 Dataset Comparison

We conducted a comprehensive comparison between our dataset and existing mathematics question generator datasets. As shown in Table 3, Our EduMath dataset demonstrates several significant advantages. First, the scale of our dataset substantially surpasses other comparable datasets in terms of problem quantity. Second, our dataset specifically focuses on high school mathematics problems that require deeper cognitive reasoning, whereas datasets like LMWP and HMWP primarily target elementary-level mathematics question generation. The generation of high school mathematics problems presents considerably greater challenges due
Figure 4: The cost refers to the average cost required to generate a question on the EduMath-CQ dataset.

to their complexity and reasoning requirements. Furthermore, our dataset features meticulous annotations performed by large language models (LLMs). We implemented a rigorous two-stage annotation process: after the initial annotation phase, we employed LLMs to verify the accuracy of the annotations, with any identified inaccuracies undergoing a re-annotation process. This makes EduMath the first and only open-source high school mathematics dataset that incorporates multi-dimensional educational objective annotations, setting a new standard for educational resource development.

A.4 Educational Metric Details

Solvable We employed the state-of-the-art large language model, DeepSeek-V3, to solve the generated questions. We then determined the solvability of each question using a majority voting approach based on self-consistency (five samples).

Pass Rate We employ Chain of Thought (COT) reasoning by inputting both educational objectives and generated questions into the large language model, enabling step-by-step analysis of whether the question meets each educational objective. Using the Self-Consistency approach, we generate 5 independent judgment samples and determine the final result through majority voting. If an question fails to meet any educational objective, it is marked as failing.

Win Rate The Win Rate metric evaluates question quality by inputting pairs of questions along with their educational objectives into the large language model to determine which is superior. The evaluation criteria include adherence to educational objectives, natural language flow, and seamless integration of context. As shown in Table 10, we provide a specific case study demonstrating this judgment process.

A.5 Implementation Details

Parameter Details. For LLM parameter settings, we maintained a consistent temperature of 0.7 across all four models to ensure output diversity. The model versions used in our testing include gpt-4o-mini-2024-07-18, Claude-sonnet-3.5-0622, GPT-4o-2024-11-20, and DeepSeek-V3. Regarding method parameters, for COT-BOT (Best of N), we generated 5 candidate outputs and selected the best result; for our proposed EPQR method, the Monte Carlo Tree Search (MCTS) parameters were configured as follows: 4 iterations, maximum depth of 3, and an exploration parameter c of 2.5 in the
Table 3: Comparison of Different Mathematics Problem Datasets

<table>
  <tr>
    <th>Datasets</th>
    <th>EduMath (Ours)</th>
    <th>GAOKAO (<i>Zhang et al. (2023)</i>)</th>
    <th>LMWP (<i>Liu et al. (2020)</i>)</th>
    <th>HMWP (<i>Qin et al. (2020)</i>)</th>
    <th>GSM8k (<i>Cobbe et al. (2021)</i>)</th>
  </tr>
  <tr>
    <td>Deep Reasoning</td>
    <td>✓</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>Objectives Annotation</td>
    <td>✓</td>
    <td>✗</td>
    <td>✓</td>
    <td>✓</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>Multi-Edu objectives</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>#Problems</td>
    <td>16348</td>
    <td>500</td>
    <td>5477</td>
    <td>5476</td>
    <td>8500</td>
  </tr>
</table>

Table 4: Number of Questions and Average Number of Concepts for Two Datasets

<table>
  <tr>
    <th>Dataset</th>
    <th># Questions</th>
    <th>Avg. # Concepts</th>
  </tr>
  <tr>
    <td>EduMath-SQ</td>
    <td>10763</td>
    <td>2.57</td>
  </tr>
  <tr>
    <td>EduMath-CQ</td>
    <td>5585</td>
    <td>2.39</td>
  </tr>
</table>

UCT formula.

Output strategy. Each iteration of the Monte Carlo Tree Search (MCTS) yields a path from the root node to a leaf node. Following PromptAgent (<i>Wang et al., 2023</i>), we select the path with the highest average question reward and then choose the question with the highest reward from that path as the final output. This strategy ensures that we identify the best question from the overall optimal search trajectory.

A.6 Baseline Details

• Chain-of-thoughts (CoT) (<i>Wei et al., 2022</i>): Prompts language models to think step-by-step before reaching final conclusions, incorporating deliberate reasoning and systematic thinking to generate more powerful and insightful answers.

• Chain-of-thoughts Best of N (<i>Wei et al., 2022</i>): Samples multiple CoT outputs and selects the best one from the generated candidates.

• ReAct (<i>Yao et al., 2023</i>): Simulates human problem-solving patterns through reasoning and action steps, enabling large language models to better understand tasks, gather information, execute operations, and correct errors, thereby significantly improving their performance on complex tasks. For the education question generation task, we adopt a thought + action approach, where the model generates a thought before producing the final question.

• Dear (<i>Xue et al., 2024</i>): A human cognition-inspired reasoning framework that builds a reasoning tree through a three-stage cycle.

It decomposes complex problems into sub-problems in the Decompose stage, generates and self-checks reasoning processes for each sub-problem in the Analyze stage, and updates parent node reasoning based on child node results in the Rethink stage, thereby enhancing large language models’ complex reasoning capabilities.

Table 5: Additional Evaluation results on datasets EduMath-CQ(%).

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">WIN RATE</th>
    <th>PASS RATE</th>
  </tr>
  <tr>
    <th>GPT-4o</th>
    <th>Claude-3.5</th>
  </tr>
  <tr>
    <td>COT</td>
    <td>38.16</td>
    <td>37.95</td>
    <td>34.78</td>
    <td>32.08</td>
  </tr>
  <tr>
    <td>COT-BON</td>
    <td>34.34</td>
    <td>41.61</td>
    <td>36.58</td>
    <td>38.37</td>
  </tr>
  <tr>
    <td>REACT</td>
    <td>42.68</td>
    <td>45.27</td>
    <td>40.72</td>
    <td>41.26</td>
  </tr>
  <tr>
    <td>DEAR</td>
    <td>41.30</td>
    <td>44.19</td>
    <td>39.59</td>
    <td>34.75</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>44.07</b></td>
    <td><b>47.11</b></td>
    <td><b>43.43</b></td>
    <td><b>45.92</b></td>
  </tr>
</table>

A.7 Human Evaluation Criteria

• 0: Incomprehensible - The question is confusing and impossible for students to understand, making it impossible to answer.

• 1: Partially Clear - Students can grasp the core idea of the question and attempt to answer, but the question still needs improvement.

• 2: Completely Clear - The question is concise, clear, easy to understand, and allows students to answer smoothly.

B Additional Results

B.1 Evaluation of Methods on More LLMs

To validate the model-agnostic nature of our framework, we conducted experiments on EduMath-CQ using GPT-4o and Claude-3.5 as backbone models,
Figure 5: Win Rate Comparison Matrix Across Different Methods

with results presented in Table 5. The experimental results demonstrate that our framework consistently outperforms the baseline across both models in terms of Win Rate and Pass Rate metrics. This consistent superior performance indicates that our framework’s effectiveness is model-agnostic and can be successfully applied across various large language models. Additionally, Claude-3.5 demonstrates the strongest performance among the four models, likely attributed to its enhanced reasoning capabilities and superior instruction-following abilities. This finding further indicates that the performance of the foundation model has a significant impact on the overall effectiveness of the framework.

B.2 Cost Analysis

We presented a comparative analysis of four models on the EduMath-CQ dataset, examining both the average cost per question generation and the corresponding pass rates. The Figure 4 demonstrates that all models performed better under the EQPR method compared to the baseline. Notably, DeepSeek achieved exceptional cost-effectiveness, attaining the third-highest pass rate at less than $0.01 per question, significantly lower than the costs of Claude-3.5 and GPT-4o. While the Claude-3.5-based EQPR solution had the highest cost, it achieved the best pass rate, with expenses still remaining under $0.1 per question. Given that the cost of human-generated questions would be considerably higher than this amount, we consider this investment to be justified.

B.3 Win Rate Comparison of Methods

To evaluate the performance of these five methods in a more fine-grained manner, we conducted pairwise comparisons on the EduMath-CQ dataset based on DeepSeek-V3’s results and calculated their respective Win Rates, as detailed in the Figure 5. The results demonstrate that our proposed method, EQPR, outperforms other methods in terms of Win Rate, highlighting its superiority. Specifically, EQPR exhibits significant advantages even when compared to strong baseline methods. For instance, EQPR achieved Win Rates of 58.5% and 62.1% against DEAR and REACT, respectively. This suggests that EQPR is more effective in guiding the model to generate high-quality questions that are both coherent and aligned with educational objectives.

B.4 case study

To better illustrate the process of question generation, critic, and revision, Table 6 presents a case study based on the learning objective of "categorizing random events." The system first generates an initial question based on predefined education objectives : “Which of the following describes a random event?...” The Critic module then evaluates the question, assigning a score of 6 and identifying several issues, such as the lack of a real-world context, failure to address impossible events, and limited cognitive depth. In response, the Reflection module proposes concrete revisions—for example, ensuring that the answer choices include random, certain, and impossible events, maintaining a single-choice format, and keeping the difficulty level moderate. Based on these suggestions, an improved version of the question and answer options is generated. This case study demonstrates how the question quality can be progressively refined to better align with instructional goals and cognitive requirements.
Table 6: Case study of an example generated by our method (Claude-3.5)

<table>
  <tr>
    <th colspan="2">Education Objectives</th>
  </tr>
  <tr>
    <th>Education Objectives:</th>
    <td>
      <ul>
        <li><b>Concept:</b> Random events</li>
        <li><b>Core Quality:</b> Mathematical abstraction, logical reasoning</li>
        <li><b>Core Ability:</b> Distinguish between random, certain, and impossible events, and make probability judgments in real-life contexts</li>
        <li><b>Context:</b> Real-life scenarios involving random events (e.g., dice rolling, lottery, baby gender) to enhance relevance and interest while testing key concepts</li>
        <li><b>Bloom’s Taxonomy:</b> Understanding</li>
      </ul>
    </td>
  </tr>
  <tr>
    <th colspan="2">Question</th>
  </tr>
  <tr>
    <th>Question:</th>
    <td>
      Which of the following describes a <i>random</i> event?<br>
      A. It will rain tonight    B. The sun will rise in the west tomorrow    C. There are 12 months in a year    D. A coin toss will definitely land heads up
    </td>
  </tr>
  <tr>
    <th colspan="2">Updated</th>
  </tr>
  <tr>
    <th>Critic:</th>
    <td>
      <b>score:</b> 6.<br>
      <b>reason:</b> Covers basic concept, but lacks real-life context; omits “impossible event”; low cognitive demand; insufficient instructional depth
    </td>
  </tr>
  <tr>
    <th>Reflection:</th>
    <td>
      <b>thought:</b> Include random, certain, and impossible options with plausible distractors; keep single-choice format and moderate difficulty; use parallel comparisons to guide classification by definition.<br>
      <b>question:</b> Which of the following describes a <i>random</i> event?<br>
      A. Rolling a fair die shows a 6.<br>
      B. The sun will rise in the west tomorrow.<br>
      C. There are 12 months in a year.<br>
      D. A tossed coin will definitely land heads up.
    </td>
  </tr>
</table>
Algorithm 1 MCTS with Reflection

Require: Initial question(state) s (root), expand width k, exploration weight w, max_iterations, depth_limit, reflection module reflection, critic module critic
Initialize root node with initial state
for \( i \leftarrow 1 \) to \( \text{max\_iterations} \) do
    path \( \leftarrow [] \)
    node \( \leftarrow \) root
    # Selection Phase
    while node has children do
        node \( \leftarrow \) argmax\(_{child \in node.children}\)(child.uct)
        Add node to path
        Update node.visited
    end while
    # Expansion Phase
    if node.depth < depth_limit then
        for \( j \leftarrow 1 \) to \( k \) do
            Generate new child through reflection:
            question \( \leftarrow \) Reflection
            child \( \leftarrow \) create_node(question, parent = node)
            Evaluate child.reward using Critic
            Add child to node.children
        end for
    end if
    # Simulation Phase
    while not terminal node do
        Select child with the highest immediate reward
        Add node to path
        Update node.visited
    end while
    # Backpropagation Phase
    cumulative_reward \( \leftarrow 0 \)
    for node in reversed(path) do
        cumulative_reward \( \leftarrow \)
        cumulative_reward + node.reward
        Update node.cum_rewards with
            cumulative_reward
        Calculate new node.Q value
    end for
end for
return the best question from the path
<table>
  <tr>
    <th>Question Generation Prompt</th>
  </tr>
  <tr>
    <td>You are an expert in high school mathematics education. You are analyzing educational objectives to design and create a multiple-choice question. Your goal is to develop a well-structured question that aligns with specific educational objectives while fostering core competencies.<br><b>Status Determination Rules:</b><br><b>concepts:</b> Ensure complete alignment with required content; Maintain logical rigor and clear progression; Consider cognitive levels<br><b>Competency Development:</b> Integrate core competencies naturally with content; Build connections between concepts<br><b>Example:</b> {few_shots}<br><b>Output Format:</b><br>{"question_design_thought": "detailed explanation of question design approach", "question": "complete multiple-choice question with options"}<br><b>Required Input:</b> Education_Objectives: {educational objectives}</td>
  </tr>
</table>

Table 7: Question generation prompt template.

<table>
  <tr>
    <th>Critic Evaluation Prompt</th>
  </tr>
  <tr>
    <td>You are an expert in high school mathematics education. Your task is to evaluate a mathematical question and its design approach based on given educational objectives. You will assess whether the question meets the educational objectives, provide a strict scoring evaluation, and analyze areas for improvement.<br><b>Scoring Scale (1-10):</b><br><b>Excellent (10):</b> - Complete alignment with Concept and competency requirements - Clear, structured design following stated approach - Deep pedagogical design fostering core competencies - Appropriate cognitive level for students<br><b>Good (8-9):</b> - Generally meets educational objectives - Minor deviations from target cognitive level<br><b>Average (6-7):</b> - Meets content requirements but lacks competency development - Cognitive level misalignment with objectives<br><b>Fair (4-5):</b> - Only partially meets content requirements - Significant cognitive level misalignment<br><b>Poor (1-3):</b> - Severe deviation from educational objectives - Does not follow design approach - Low quality question<br><b>Failing (0):</b> - No connection to educational objectives - Completely unrelated to goals and design approach<br><b>Key evaluation points:</b><br><b>Concept Alignment:</b> - Check for complete coverage of required content - Assess logical structure and progression - Evaluate cognitive level appropriateness<br><b>Competency Development:</b> - Analyze integration of core competencies - Assess effectiveness in building understanding<br><b>Output Format:</b><br>{"direction": "detailed analysis of weaknesses, and improvement suggestions", "score": numerical score 1-10}<br><b>Required Input:</b> Education_Objectives: {educational objectives}; Question: {current question}</td>
  </tr>
</table>

Table 8: Critic evaluation prompt template.
<table>
  <tr>
    <th>Reflection Prompt</th>
  </tr>
  <tr>
    <td>You are an expert in high school mathematics education. Your task is to analyze and optimize a math question and its design approach based on given educational objectives, previous feedback, and the question’s evolution history. Your goal is to refine both the question design approach and the question itself to better meet educational objectives while maintaining high quality.<br><b>Key Constraints:</b><br>- All questions must have valid solutions - Modifications should go beyond numerical changes - Changes must align with existing objectives - No new concepts or competencies can be added<br><b>Analysis Points:</b><br><b>Previous Question Analysis:</b> - Review strengths and weaknesses identified - Understand suggested improvements - Study evolution of previous versions<br><b>Optimization Strategy:</b> - Address identified weaknesses - Maintain existing strengths - Enhance alignment with objectives - Improve question quality<br><b>Educational Alignment:</b> - Verify concept coverage - Check competency development<br><b>Output Format:</b><br>{"thought": "detailed explanation of optimized design approach", "question": "complete optimized question"}<br><b>Required Input:</b> Education_Objectives: {educational objectives} Question: {current question} Direction: {current feedback and suggestions} Trajectory_thoughts: {history of previous versions}</td>
  </tr>
</table>

Table 9: Reflection prompt template.
Table 10: Example of the Input and Output for the Education Objectives Evaluation Module

<table>
  <tr>
    <th colspan="2">Education Objectives</th>
  </tr>
  <tr>
    <th>Education Objectives:</th>
    <td>
      <ul>
        <li><b>Concept:</b> Permutation and Combination, Counting Principle</li>
        <li><b>Core Quality:</b> Mathematical Operations, Logical Reasoning</li>
        <li><b>Context:</b> School Course Scheduling (combining combinatorial mathematics with practical school management)</li>
        <li><b>Core Ability:</b>
          <ul>
            <li>Handle complex permutation and combination problems.</li>
            <li>Apply classification counting and step-by-step counting techniques.</li>
            <li>Solve practical problems using enumeration and classification addition principles.</li>
          </ul>
        </li>
        <li><b>Bloom’s Taxonomy:</b> Application</li>
      </ul>
    </td>
  </tr>
  <tr>
    <th colspan="2">Question Pairs</th>
  </tr>
  <tr>
    <th>Question 1:</th>
    <td>A school has 3 courses to assign for the semester, and each course must be taught by one teacher. There are 2 available teachers to choose from for each course. How many possible ways can the teachers be assigned to the courses?</td>
  </tr>
  <tr>
    <th>Question 2:</th>
    <td>A school needs to arrange courses for the semester. There are 5 courses, and each course needs to be assigned a teacher. Assume each course can be taught by one of 3 teachers, and each teacher can teach at most two courses. How many ways can the courses be assigned to teachers?</td>
  </tr>
  <tr>
    <th colspan="2">Evaluation</th>
  </tr>
  <tr>
    <th>Better Question:</th>
    <td>2</td>
  </tr>
  <tr>
    <th>Reason:</th>
    <td>Question 1 only covers 'the basic counting principle of assigning teachers to courses' without considering any constraints. It does not address the permutation aspect, which is an important part of the core educational objectives. On the other hand, Question 2 is more complete as it covers both permutation and combination, as well as the counting principle, by introducing a constraint (each teacher can teach at most two courses). This makes Question 2 a better fit for evaluating the student's ability to apply counting techniques and solve combinatorial problems. Therefore, Question 2 is a more suitable and well-rounded problem for this purpose.</td>
  </tr>
</table>
<table>
  <tr>
    <th>Pass Rate Evaluation Prompt</th>
  </tr>
  <tr>
    <td>Consider the following task as a mathematics education assessment assignment. You will act as a senior mathematics education expert, rigorously evaluating a given math problem and its associated educational objectives. You will assess based on the following six key dimensions, checking each item meticulously. If any item does not meet the requirements, the overall assessment result will be marked as a failure (outputting 0):

<b>Key Evaluation Dimensions:</b><br>
<b>1. concept Coverage:</b> - Verify complete coverage of all required concepts<br>
- Any omission or introduction of unmentioned points is non-compliant<br>
<b>2. Bloom's Taxonomy Level:</b> - Analyze cognitive level alignment with objectives - Must neither exceed nor fall short of target level<br>
<b>3. Student Problem-Solving Skills:</b> - Confirm comprehensive assessment of all targeted skills<br>
<b>4. Mathematical Core Competencies:</b> - Evaluate cultivation of: * Logical reasoning * Mathematical operations * Spatial visualization * Data analysis * Mathematical modeling * Mathematical abstraction<br>
<b>5. Rigor Requirement:</b> - Maintain objectivity and rigor throughout evaluation - Any non-compliance results in direct failure (0)

<b>Output Format:</b><br>
{ "reason": "Detailed explanation of the reasoning and process behind the evaluation", "pass_rate": 1 or 0 }

<b>Required Input:</b> Education_Objectives: {educational objectives} Question: {question}</td>
  </tr>
</table>

Table 11: Pass rate evaluation prompt template.
<table>
  <tr>
    <th>Win Rate Evaluation Prompt</th>
  </tr>
  <tr>
    <td>As a senior mathematics education expert, please rigorously evaluate and compare the following question pair. In the evaluation process, analyze each question based on the following dimensions and determine which question better meets the educational objectives.<br><b>Evaluation Dimensions:</b><br><b>1. Completeness of Concept Coverage:</b> - Analyze coverage of required concepts - Check for missing or redundant points<br><b>2. Matching of Cognitive Levels:</b> - Assess alignment with specified cognitive level - Verify appropriate goal alignment<br><b>3. Relevance to Ability Development:</b> - Confirm effective training of specified abilities - Verify alignment with outlined requirements<br><b>4. Development of Mathematical Literacy:</b> - Analyze contribution to mathematical literacy development<br><b>5. Scientific Design of the Structure:</b> - Evaluate reasonableness of question structure - Assess organization and guidance quality<br><b>6. Text Clarity and Coherence:</b> - Assess clarity and conciseness of wording - Evaluate effectiveness of problem-solving communication<br><b>Output Format:</b><br>{ "better_question": 1 or 2, "reason": "Detailed evaluation reasons, explaining why the selected question is better and specifying which dimension(s) show superior performance." }<br><b>Required Input:</b> Education_Objectives: {educational objectives} Question Pair: {Question pair}</td>
  </tr>
</table>

Table 12: Win Rate evaluation prompt template.

Reviewer paper summaries: This paper proposed planning based framework for generating concept-based mathematical questions that meets the objectives of education. They define some objectives of education for this purpose. They curated a large dataset (EduMath) from mock exam papers and annotated them with these objectives. Their proposed framework has a critic module, reflection module, and a MCTS planner module. Critic module verify the alignment of generated questions with educated objectives. Reflection module takes the feedback from critic module and reflect upon to fix errors. MCTS planner module helps in searching the vast exploration space and generate questions with education objectives. They compare their proposed framework with previous reasoning frameworks such as COT, ReAct, etc. and showed that proposed framework does better on the EduMath dataset.

- The paper automatically labels a mathematical question datasets for 5 dimensions of educational objectives.
- The paper proposes a novel Actor-Critique style mathematical question improvement method. 
- It experiments with 5 LLM-based educational question generation approaches starting from educational objectives.

Comments after Author Discussion:
The authors responded. Based on the details and availability of the dataset, I updated my scores.

On the other hand, I agree that "using large language models as evaluators has become a mainstream trend in research", but LLM evaluators have to be validated by human evaluators. Pass rate ratio is not a sufficient validation of LLM evaluators. After a careful human validation of the LLMs (next cycle), I'll be happy to accept the paper.

The authors investigate the quality of questions generated based on educational objectives. To do this, they created a database of already existing test questions and automatically assigned educational objectives to each of them.

The main contribution of this article is the planning-based educational question generation method. Based on an educational objective, it generates and iteratively refines a question with LLMs while using Monte Carlo Tree Search for planning.

The authors suggest automatic evaluation metrics that they ran on the whole task. They also manually evaluated a subset of the questions.
Summary of strengths: 1. Authors curated a EduMath dataset that have large number of mathematical questions aligned with educational objectives.
2. Authors proposed a new framework to generate such questions with critic/reflection/planner module.
3. Proposed framework performs better than the previous reasoning frameworks such as COT, ReAct, etc on EduMath dataset.

- EduMath is a resource which is valuable for educational NLP research
- The proposed EQPR process is novel, interesting and well-motivated by human educators' question writing processes.
- Experimental results are convincing, that EQPR outperforms the four very strong baseline approaches.

The fact that the authors created a dataset, as well as implemented a solution and an evaluation system for the defined task-specific evaluation metrics that surpass the standard baselines in both its complexity and its effectiveness shows a strong commitment to this issue. The idea of combining the iterative refinement and the Monte Carlo Tree Search is interesting. They applied strong baselines, and the human evaluation of 100 - 100 examples showed the strength of their proposed method.
Summary of weaknesses: 1. Most experiments were done using DeepSeek LLM. Authors must explore some other LLM as well for more generalizability.

1. LLMs are used as automated labeller and evaluators almost everywhere in the paper and  their reliability is not validated. The only exception is WinRate in Table 2 and human WinRates achieved by the approaches do not follow the same ranking as Table 1 automatic WinRates.
- Section 5 presents "95.2% annotation accuracy". How was annotation accuracy measured?
- What is the reliability of the automated LLM evaluators?

2. The main experimental results compare a generated educational question against a gold standard question for the same educational objectives. What is the motivation for trying to generate the same questions?
Especially WinRate counts the ratio the generated question outperforms human written questions. Other educational question quality metrics (like faithfulness, correctness, engagement) would be more useful. I prefer the systems pairwise comparison (Figure 5 in the appendix) over Table 1 as a support of EQPR outperforms other methods.

3. According the human evaluation clarity scores, even the baselines have only 13 out of 100 non-perfect questions. I think that the few numbers of example EQPR could fix does not worth the computational efforts. What is your take on this?

4. The paper does not contain any examples. What type of questions are generated (multiple-choice?)? In English or other languages as well? A few examples of questions generated by various methods would help to understand the complexity of the problem and the evaluation metrics as well.

The authors report high annotation accuracy on their EduMath dataset but they didn’t share any information on how they calculated that score.

The authors only mention the BLEU, ROUGE, and other automatic comparison-based metrics for evaluating question generation, but there are previously applied metrics that evaluate properties of the question like clearness, complexity, consistency to predefined Bloom category, etc. for example in the following papers:

Luo, Haohao, Yang Deng, Ying Shen, See-Kiong Ng, and Tat-Seng Chua. "Chain-of-exemplar: enhancing distractor generation for multimodal educational question generation." ACL, 2024.
Chen, Yuyan, Chenwei Wu, Songzhou Yan, Panjun Liu, and Yanghua Xiao. "Dr. Academy: A Benchmark for Evaluating Questioning Capability in Education for Large Language Models." In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3138-3167. 2024.

Another downside of the proposed method is its cost. The continued improvement of the questions requires a lot of LLM calls, and the second-best system on human evaluation (REACT) has a significantly lower price. On the other hand, if the price is not an issue, the authors' method achieved the best results.
Comments, suggestions and typos: NA

- please add examples to the paper, especially for the evaluation of a question by EQPR would be helpful
- a broader list of actions suggested by the Reflection module would be interesting to see
- a more detailed (than Section 6.4) analysis and discussion would be great

No further suggestions.
Overall assessment scores: 4

2

4