Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation
Authors: Zhenwen Liang
Wenhao Yu
Tanmay Rajpurohit
Peter Clark
Xiangliang Zhang
Ashwin Kalyan
Venue: EMNLP 2023 Main (2023)
Keywords: Mathematical Reasoning
Large Languague Models
Customized Learning

Abstract: In this paper, we present a novel approach for distilling math word problem solving capabilities from large language models (LLMs) into smaller, more efficient student models. Our approach is designed to consider the student model's weaknesses and foster a tailored learning experience by generating targeted exercises aligned with educational science principles, such as knowledge tracing and personalized learning. Concretely, we let GPT-3 be a math tutor and run two steps iteratively: 1) assessing the student model's current learning status on a GPT-generated exercise book, and 2) improving the student model by training it with tailored exercise samples generated by GPT-3. Experimental results reveal that our approach outperforms LLMs (e.g., GPT-3 and PaLM) in accuracy across three distinct benchmarks while employing significantly fewer parameters. Furthermore, we provide a comprehensive analysis of the various components within our methodology to substantiate their efficacy.

Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation
Zhenwen Liang x1 Wenhao Yu², Tanmay Rajpurohit
Peter Clark Xiangliang Zhang', and Ashwin Kaylan
'University of Notre Dame, zliang6ênd.c edu
Tencent AI Seattle Lab
Georgia Institute of Technology
Allen Institute for AI, ashwinkveallenai. org
A Abstract
Comprarisa of Our Method with Baselines
80
Fine Baselines
In this paper, we present novel approach for distilling math word problem solving capabili- ties from large language models (LLMs) into smaller, more efficient student models. Our approach designed to consider the student model's weaknesses and foster lailored learn- ing experience by generating targeted exercises aligned with educational science principles, such as knowledge tracing and personalized learning. Concretely, we let GPT-3 be math tutor and two steps iteratively: I) assess- ing the student model's 's current learning status on a GPT-generated exercise book, and 2) proving the student model by training it with tailored exercise samples generated by GPT-3. Experimental results reveal that our approach outperforms LLMs (e.g., GPT-3 and PaLM) accuracy across three distinct benchmarks while employing significantly fewer parame- ters. Furthermore, we provide comprehensive analysis of the various components within our methodology substantiate their efficacy.
Our Method
100
Parameters
1011
#/erameters
100
20-
10"
10-
10
GF Our
13.0M
15.4M
5408
377M
Figure 1: Accuracies vs model sizes for representative baselines and our approach on SVAMP datasct. Our method achieves competitive performance with LLMs with significantly fewer parameters.
education, decision-making, and more (Mukherjee and Garain, 2008). The format of a typical MWP involves a textual description of a problem scenario, which needs to be translated into a mathe- matical expression (typically an equation) that can be solved to obtain the answer. MWP solving presented task for artificial intelligence several decades ago (Fletcher, 1985). Previous fine-tuned methods usually apply Seq2Seq models (Wang et al., 2017, 2018; Xie and Sun, 2019; Zhang et al., 2020; Liang et al., 2022a). In recent years, large language models (LLMs) such as GPT-3 (Brown et al., 2020) and PaLM (Chowdhery et al.. 2022) cxhibit strong reasoning ability with the help of chain-of-thought (CoT) prompting (Wei et al., 2022b; Wang et al., 2023; Chen et al., 2022), which achieves striking performance on MWP solving and outperforms fine-tuned state-of-the-arr (SOTA) solvers by a large margin.
1 Introduction
Math word problems (MWPs) are an essential aspect of mathematical education and a critical skill to develop for individuals across various domains in life (Hegarty et al., 1995). MWP solving requires the ability to comprehend natural language, extract relevant information, and perform mathematical reasoning solve the given problem. Research in MWP solvers has gained considerable interest due to the ubiquity of such problems in daily life, ranging from finance and scheduling to engineering and science (Koedinger and Nathan, 2004). Developing AI agents capable of solving MWPs demonstrates an advanced understanding of the interplay between natural language processing and mathematical reasoning, paving the way for practical applications in
Despite their remarkable strides in solving MWPs, the substantial number of parameters LLMs results computational inefficiency, and necessitates API availability for reproducibility. To
This work was done when Zhenwen L Liang and Wenhao Yu were intems at the Allen Institute for AI (A12).

---

mitigate these issues, a natural solution is distilling the knowledge from LLMs (i.e. teacher models) into smaller, more efficient models (i.e. student models). The majority of prior research has em- phasized using the "explanation" component of the CoT approach as the distilled knowledge (Ho et al., 2022; Li et al., 2022a; Shridhar et al., 2022b; Magis- ter et al., 2022). Nonetheless, these methodologies exhibit certain limitations. Firstly, small student models may struggle understand CoT explana- tions, potentially impeding their learning efficacy. Evidence from (Wei et al., 2022b,a) indicates that only exceptionally large language models possess the capability to perform CoT during reasoning. As result, many student models (Ho et al., 2022; Li et al., 2022a; Magister et al., 2022) trained on explanations do not attain satisfactory accuracy in MWP-solving tasks. Secondly, those distillation processes lack feedback from the student model to the LLM teacher, and thus, neglect to consider the knowledge acquisition state of the student.
ally by exercises, thereby facilitating the genera- tion of tailored educational experiences. Person- alized learning also of vital importance (Hattie and Timperley, 2007: Grant and Basye, 2014). Be- ing cognizant of the student model's learning sta- tus ensures the optimally designed exercises are generated to address the model's specific weak- nesses. By monitoring the learning progress, our proposed method can dynamically adapt the stu- dent model's evolving knowledge state, fostering more effective learning outcomes. In our method, we integrate knowledge tracing and learning status into the distillation process to establish a robust connection between the LLM teacher and the stu- dent model, yielding a more interactive and cus- tomized learning experience. Consequently, this ap- proach substantially enhances the student model's problem-solving capabilities. As illustrated in Fig- ure 1, our knowledge distillation approach achieves competitive accuracy on the SVAMP dataset, but employs significantly fewer parameters compared to state-of-the--ar LLMs, such as GPT-3+CoT and PaLM+CoT with few-shot prompting (Wei et al., 2022b).
In this paper, we also concentrate on harness- ing the strengths LLMs to coach smaller, more efficient MWP solvers, but introduce a novel method for addressing the limitations of previ- ous approaches: Customized Exercise for MAth Learning (CEMAL). We reframe the problem by shifting focus from providing explanations for existing exercises (i.e., training set) to identifying the student model's learning needs and general- ing new exercises tailored to them. Our CEMAL methodology offers a new perspective on knowl- edge distillation with LLMs by focusing on cus- tomized exercise generation, based on the student models' current learning state. This enables more effective and adaptive learning, addressing limita- tions in prior approaches that rely heavily on gen- erating more high-quality explanations to existing datasets. This approach offers several advantages: (I) does not impose CoT ability requirements on small models, allowing them to learn more effec- tively, (2) it takes into account the learning status of the student model during training.
Our contribution can be summarized follows:
We propose a novel method named CEMAL that utilizes LLMs to generate additional data in the form of targeted practice problems, ad- dressing the student model's 's weak areas.
Our approach is evaluated on multiple MWP datasets, including both in-distribution (ID) and out-of-distribution (OOD) tests (Koncel- Kedziorski et al., 2016; Miao et al., 2020; Pa- tel et al., 2021). We show that our method is significantly effective in improving student models under the OOD setting.
The experimental results demonstrate that our method achieves state-of-the- the-art accuracy, sig- nificantly outperforming fine-tuned baselines. Notably, the student model trained with our method even surpasses LLMs with fhw-shot CoT prompting, despite having significantly fewer parameters.
In fact, our approach CEMAL seamlessly aligns with two fundamental tasks in educational sci- ence: knowledge tracing and personalized learn- ing. Knowledge tracing pertains to monitoring and modeling a student's evolving knowledge state over time (Corbett and Anderson, 1994; Abdelrahman et al., 2023). This process enables the identifica- tion of a learner' 's strengths and weaknesses, usu-
Related Work
2.1 Math Word Problem Solving
After many years of research on rule-based algo- rithms (Hosseini et al., 2014; Mitra and Baral, 2016) and semantic parsing methods (Shi et al.,

---

2015; Huang al., 2017), deep learning has become the predominant technique for solving MWPs, thanks to its superior performance and better generalization ability. Deep neural solver (DNS) (Wang et al., 2017) was among the first to apply Seq2Seq models with RNN encoder and de- coder for MWP solving, and subsequent research has mainly explored different structures of RNN- to-RNN solvers (Wang et al., 2018; Liu et al., 2019a; Xie and Sun, 2019; Li et al., 2019; Zhang et al., 2020; Liu et al., 2021). More recently, pre- trained language models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b) have demonstrated remarkable language understanding abilities, leading researchers to replace MWP en- coders with these models (Li et al., 2022b; Huang et al., 2021; Shen al., 2021; Patel et al., 2021; Liang et al., 2022a), resulting in significant accu- racy improvements. Recently, LLMs have shown great success in MWP solving. with much superior accuracy compared to fine-tuned baselines, simply by being provided with few CoT examples of the bobblem-solving processes. Interestingly, re- searchers also found that LLMs can reason with zero-shot prompts such as "Let's think step by step" (Kojima et al., 2022). Currently, numerous studies have been conducted improve the performance of LLMs, including recent works by (Wang et al., 2023; Chen et al., 2022; Zhou et al., 2023; Diao et al., 2023; Liang et al., 2023).
AugESC (Zheng et al., 2022) leverages the GPT-J (Wang and Komatsuzaki, 2021) model and utilizes publicly available dialog posts to trigger conversa- tions on various topics. Then, West et al. (2022) em- ploys a generalized LLM to create common sense knowledge graphs, while WANLI (Liu et al., 2022) combines LLM-generated examples with human evaluation to establish diverse natural language in- ference datasets. Additionally, ZeroGen (Ye et al., 2022b) proposes a zero-shot learning approach by generating datasets using pre-trained LLMs, Pro- Gen (Ye et al., 2022a) uses the quality of generated samples as feedback to LLMs to improve the data generation, and Shao et al. (2023) feeds chain-or thought demonstrations to LLMs and targets gener- ating more exemplars for in-context learning. In the domain of MWP solving, several stud- ies have also been conducted with the objective of distilling the reasoning capability of LLMSs into smaller solvers by employing stain-of-thought ex- planations. (Ho et al., 2022) introduces Fine-tune- COT, which uses LLMs to generate reasoning step instances, subsequently facilitating the fine-tuning of smaller models. (Li et al.. 2022a) explores three explanation generation strategies and incorporates them into multi-task learning framework tailored for compact models. (Magister al., 2022) as- sesses the efficacy of chain-of-thought explanations in the training of a small model across three dis- parate tasks, namely arithmetic reasoning, com- monsense reasoning, and symbolic reasoming. Fur- thermore, (Shridhar et al., 2022b) presents Decom- positional Distillation, an approach that segments the problem into subproblems to enhance smaller models' performance.
In this paper, we leverage the great success of LLMs to facilitate the training of the student solver. Our approach utilizes an efficient fine-tuned solver as its backbone and yields even superior perfor- mance to LLMs, further pushing the limit of deep learning methods in MWP solving.
In contrast to these existing works, our proposed knowledge distillation approach in MWP solving is unique in that it does not focus on the chain-of- thought explanation and it takes into account the learning status of the student model and generates exercises that tailor to the specific weaknesses of the student. Our approach bridges the gap between knowledge distillation and data augmentation in the context of math word problem solvers, allowing student models to improve their problem-solving capabilities more effectively.
2.2 Large Language Models for Knowledge Distillation and Data Generation
In recent years, there has been a surge of interest in knowledge distillation from LLMs to smaller mod- els. Due to the unavailability of model structure for LLMs, their application is often limited to prompt design and subsequent data generation. There- fore, data generation has become an integral part of knowledge distillation in the context of LLMs. Several studies have investigated the potential of LLMs in knowledge distillation and data genera- tion. For instance, PromDA (Wang et al., 2022) applics prompt-based data augmentation to low- resource natural language understanding tasks, and
3 Approach
3.1 Problem Definition
Our objective is to train a student Math Word Prob- lem (MWP) solver with the assistance of large lan-

---

Algorithm 1 Training Framework
4. Repurt to the teacher
Input: Training Set Terain. Exercise Generation Function EG. Exercise Book X, Student Solver Parameters:
Identity know State
Probability Threshold X, number of generations
m,
Student Modk!
Evaluate book
Pufore mininng Generuze exertise borks
1: for Each W-A puir in Terain do
b Before Training >> Initial
2: Terain + Terain + EG(W, A.
(6). Learning on custom ized exercises.
COOO
Augmentation by m times
3: end for
LLM
Teacher
4: for Each W-A pair in Tirats do & Generate an exercise book with times the size of the training set
5: X' t EG(W, A,
Generate sustomized training
exercives
6: end for
7:
while Training do
Training starts
8: for Each W-A pair in Ternio do Train Solver S with W-A end for
Figure 2:
This figure shows the overall iterative frame-
9:
work of CEMAL. After round of training, the stu- dent, which a small MWP solver, is evaluated by exer-
10:
11:
for Each W-A pair X do
cises provided by an LLM teacher. Subsequently, LLM generates customized exercises that larget the student's s knowledge state and weaknesses, thereby facilitating a customized improvement in their overall performance.
12:
if S cannot solve W' then
13:
U Uniform(0,,1
14:
p X then
15:
Tirain
Teraio EG(W,
else
Randomly sample W À pair from X Tirain < Terain + EG(W, AA
18:
guage models (LLMs). The student MWP solver
19:
end
takes a textual description W input and pro-
20:
end
21:
end for
duces an equation the solution process to arrive at the final answer. We represent quantities in W and using place- holders following number mapping (Wang et al., 2017), which unifies the representation of quan- tities in different MWPs. Specifically, W and A do not contain the actual values of quantities, but are denoted as N1, N2,..., Nk. where N; refers to the i-th number, and k is the maximum number of quantities in W and A. This approach offers two advantages: (1)
output, which indicates
22:
end while
problems that the solver is unable to solve, which reflect the knowledge state of the student that time. Subsequently, we generate eustomized exer- cises and incorporate them into the original training set Trrain to update Specifically, we generate exercises for every source MWP (the details of exercise generation are presented in section 3.4). This iterative process allows for the continuous ex- pansion of the training set and leads to enhanced performance of the student solver. Furthermore, control the proportion
widely used data pre- processing technique in MWP solving that unifies the representation of quantities and reduces the vo- cabulary size, and (2) during exercise generation, where the goal is to generate MWP variants, num- ber mapping prevents variants generated solely by changing the values of quantities.
introduce threshold of targeted generation compared to random gener- ation. detailed description of can be found in Scction 3.5. The pscudo-code of our approach is lo- cated in Alg. 1. This framework closely mimics the knowledge tracing method in human learning and thus holds promise for enhancing the effectiveness of future educational practices.
3.2 Training Workflow
Our proposed training workflow an iterative and progressive framework that integrates L.LMs into the training procedure of student MWP solvers as shown in Figure 2. Before training, we augment the training set Terain by generating more problems than the original training set, enlarge the training set at the beginning push the limit of the student solvers (lines 1-3 Alg. 1). We also generate exercise book (lines 4-6 in Alg. I). During training,
3.3 Backbone of Student Solver
In this paper, we employ Seq2Seq model with the Goal-driven Tree-based Solver (GTS) (Xie and Sun. 2019) our decoder, which has been widely applied MWP solving and shown to outperform Transformer decoders (Lan et al., 2022). For the encoder part, we use three different backbones LSTM (Hochreiter and Schmidhuber, 1997).
n times more problems to formulate
we perform model validation on X' and select the

---

RoBERTa--bse and RoBERTa-large (Liu et al., 2019b) to demonstrate the backbone-agnostie na- ture of our approach.
The exercise problems generated by LLMs may be of low quality, in a wrong format, or repetitive (a case study including correct and incorrect exam- ples is presented in Section 4.7). To alleviate the impact of these low-quality exercise problems, we filter out about 30% of problems with the wrong formats during the generation process by checking whether the generated problem is repetitive and the generated answer is valid mathematical equation.
3,4 Exercise Generation
Our method generates targeted exercises and their answers based on given source problem-answer pair (W and A), which is similar data augmen- tation. Following previous paper on analogical reasoning (Liang et al., 2022b), the generation of exercise targets to reach two kinds of analogies to W and A - problem analogy and solution analogy- that help models better understand MWPs.
3.5 Targeted vs. Random Generation
The generation of targeted exercises to address MWPs that the student model has previously failed to solve significantly improves the model's capacity to process problem and answer analogies. To eval- uate the efficacy of this approach, it is insightful to consider baseline generation strategy, namely random generation (referenced in lines 17-18 of Alg. 1). We introduce a probability threshold À to control the employment of generation strategy when augmenting the training set (lines 13-18 18 in Alg. 1). X =0 favors random generation, whereas À =1 mandates the utilization of targeted generation. A detailed analysis the impact of X can be found in Section 4.4.
For problem analogy, we generate problems with similar problem descriptions but different solutions. Our method perturbs not only questions but also considers the context, and we prompt LLMs with few-shot examples to generate MWP variants. In contrast to previous studies that used human anno- tators (Patel et al., 2021; Yang al., 2022), our approach automated and scalable. For solution analogy, we generate problems with different key- words in the problem description but a similar solu- tion to the source MWP. Interestingly, we find that LLMs can achieve this generation in a zero-shot manner by simply prompting them to generate prob- lems similar to the source problem. Our prompt design is located in the appendix.
Experiments
4.1 Datasets
This exercise generation is deployed at two dif- ferent steps in our proposed training framework. First, we generate an exercise book by augmenting the entire training set (lines 4-6 in Alg. 1), which serves a validation set to identify the weaknesses of the student solver. We do not directly use the training set validate the student solver because we want a more diverse validation set to compre- hensively evaluate the student model. Additionally, the solver may memorize the MWPs in the training set instead of fully understanding them, as previous research has found that slightly perturbed training MWPs can cause a solver to fail (Liang and Zhang, 2021). Therefore, validation the exercise book, which contains many variants of the training set, provides a more robust evaluation.
MAWPS The MAWPS dataset (Koncel- Kedziorski et al., 2016) an aggregation of 2,373 English Math Word Problems (MWPs) from various sources, including AddSub (Hos- scini et al., 2014), SingleOp (Roy et al., 2015), MultiArith (Roy and Roth, 2015), SingleEß et al., 2015), and SimulEq (Kushman al., 2014). We employ 5-fold cross-validation for the evaluation on this dataset.
ASDiv-a ASDiv (Miao et al., 2020) is an English MWP dataset designed to exhibit more diverse range of language patterns and problem types, com- prising 2,305 MWPs. In accordance with prior studies (Patel al., 2021; Lan al., 2022), we se- lect the arithmetic subset, ASDiv-a, which contains 1,218 MWPs and utilizes a 5-fold cross-validation method for evaluation.
Secondly, after identifying the problems that the student solver cannot solve from the exercise book, we use them as the source to generate customized exercises and add them the previous training set (lines 12-16 in 1). In this way, the training set grows progressively and covers more knowledge, leading to a stronger student MWP solver.
SVAMP The SVAMP dataset is test-only dataset (Patel al., 2021), consisting of 1,000 En- glish MWPs generated by introducing challeng- ing variations to existing problems. We adopt the two evaluation settings proposed in (Patel et al.,

---

MAWPS (ID)
ASDiv-a (ID)
SVAMP (ID)
SVAMP (OOD)
Prior best (Fine-tuning)
02.0/121M
82.2/114M'
65.0/144M²
47.3/121
Prior best (Knowledge Distillation)
20.7/6.7B*
Few-shot CoT (Wei et al., 2022b)
93.3/540B
93.1 /
79.0/510B
79.0/540B
LSTM (20M)
82.6
71.4
45.0
30.8
Without
Base (144M)
88.5
81.2
69.2
11.0
CEMAL
Large (377M)
90.4
87.6
78.5
49.5
LSTM (20M)
92.0
86.9
67.1
53.4
CEMAL
Base (144M)
93.9
90.9
81.5
68.6
(Our Solvers)
Large (377M)
94.7
93.3
85.4
76.4
Table 1: We compare the accuracy and number of parameters on benchmarks the format of (accuracy/number of parameters). Prior best baselines the following. (Jie al., 2022), b: (Patel et al., 2021), (Magister et al., 2022), d: (Ho et al., 2022). On each dataset, the best performance is bolded and the second best is in italics. This accuracy calculated on the ASDiv-a subset out of ASDiv based on the results in https: gi thub. com/ jas/wjas http/chhanoof--oee thouhhttt wrrigghtt-rompttii proptingg ID denotes the in-distribution test and OOD denotes the out-of-ditrriuttion test.
2021). The first setting employs 5-fold cross- validation approach the 1,000 MWPs, incorpo- rating MAWPS Kkoncel-Kedziores et al., 2016) and ASDiv-a (Miao et al., 2020) as additional training data for each fold. In the second setting, MAWPS and ASDiv-a serve as the training set, while SVAMP is used as the testing set.
RTX A6000 graphics card, implemented in Python using the Py'Torch framework. The training performed for 50 epochs with batch size of 16, us- ing the AdamW (Kingma and Ba, 2015; Loshchilov and Hutter, 2018) optimizer with an initial learn- ing rate of 8e-6 for the pre-trained model part (i.e. RoBERTa) and 5e-4 for the GTS decoder, which was halved every 20 epochs. The weight decay during training was set to 0.01, and dropout rate of 0.25 was applied the decoder to prevent over- fitting. The training set will be augmented during epoch 10 and epoch 20, i.c., lines 8-18 in Alg. will only executed at epoch 10 and 20. m and n in Section 3.2 are set to 20 and 4, respectively, with the aim of pushing the limit small math word problem solver accuracies and achieving higher per- formance. This is orthogonal to the contribution of this paper because proposed customized gener- ation strategy and exercise book are demonstrated to be effective Sections 4.4 and 4.5, Although we understand that further increasing the occurrences of validation and augmentation will further boost our accuracy, we limit their magnitude for better efficiency and lower cost on API calls. During val- idation, we generate zero-shot and 2 few-shot similar problems for each source problem in the exercise book, therefore & 4 in Alg. 1.
Evaluation Setting We categorize the evalua- tions on MAWPS, ASDiv-a, and the first setting on SVAMP as in-distribution (ID) tests, as they all involve 5-fold cross-validation on a specific dataset, Conversely, the second setting on SVAMP is con- sidered -itt---diittiittion (OOD) test, given that the training and testing sets originate from dis- tinct sources.
4.2 Implementation
The backbone of our pre-trained encoder is RoBERTa model (Liu et al., 2019b). For LLM, the ChatGPT gpt-3.5-turbo API to perform problem generation. To encourage more diverse generation, we set the temperature 1.25. All the experiments in this paper can be conducted with a cost lower than 100 dollars on OpenAl API calls. For evaluation, we use the accuracy of answer value our evaluation metric following all the baselines. Since the answer value can be calculated from dif- ferent equations, it is more reasonable to check if the value correct, rather than checking the clement-wise equivalence of generated equation and the corresponding ground truth equation.
4.3 Comparison with Baselines
In order to demonstrate the effectiveness of our knowledge distillation approach, we compare our method with several strong baselines, including the
We conducted all experiments using an NVIDIA

---

In- In-Distribution
Training set as exercise book
Same size as training set
Dataset
Backbone
n=
Random
Half Target
GTS
77.5
77.8
79.0
Base Large
54.6
59.8
60.4
65.2
ASDiv-a
RoBERTa-bees base
84.3
85.2
84.6
64.7
69.8
70.2
72.7
RoBeRTaallrge
90.1
90.3
90.6
Performance comparison of using training set and different sizes of exercise book for validation SVMP dataset.
Table 3:
GTS
89.2
88.9
89.3
MAWPS RoBERTa-base
90.3
90.3
91.0
RoBERTa-large
91.4
92.3
92.8
GTS
47.7
49.6
50.3
4.4
Analysis on Generation Strategy
SVAMP RoBERTa-base
72.1
72.8
73.3
RoBe
80.8
79.9
80.9
The performance of student solvers can be signif- icantly impacted by the generation strategies em- ployed to create the exercise problems. In this anal- ysis, we explore the effectiveness of three different generation strategies: random, half, and target (us- ing threshold values of X = 0, 0.5, 1, respectively) three datasets, in both in-distribution and out- of-distribution settings. Our goal is identify the best strategy for maximizing the student model's 's performance. We remove the initial augmentation (setting m = 0) and lower the number of genera- tions in this analysis to improve the efficiency of the experiments. Therefore, the results presented in Table 2 differ from our best results in Table 1. Our analysis reveals that the targeted generation strat- egy mostly outperforms the other two strategies. This suggests that our proposed targeted generation is an effective approach for identifying the weak areas of student solvers and improving them in a customized way.
nut-of-Distribution
Dataset
Backbone
Random Half Target
GTS
33.6
36.3
38.2
SVAMP RoBERTa basse
50.1
50.8
55.3
RoBERTa-large 62.9
63.1
65.0
Table Results of different problem generation strate- gies three datasets under in-distribution and out-of- distribution (OOD) testing. Boldface indicates the best result among the three strategies.
best fine-tuned baseline, the best LLM-enhanced knowledge distillation baseline, and few-shot chain- of-thought (CoT) prompting (Wei et al., 2022b), using the MAWPS, ASDiv-a, and SVAMP datasets. The results presented in Table 1 show that our approach outperforms all the baselines on the MAWPS and ASDiv-a datasets, achieving 94.7% and 93.3% solving accuracy, respectively. For the SVAMP dataset, our approach outperforms the best LLM-enhanced knowledge distillation base- line, achieving 85.4% accuracy on the SVAMP (ID) dataset, which is significant improvement over the prior best accuracy of 65.0% achieved by fine-tuning. On the SVAMP (OOD) dataset, our approach achieves a solving accuracy of 76.4% which lower than CoT-based LLMs, but much higher than the fine-tuned baselines. We also show the original performance of backbone solvers with- out any additional exercises. To obtain best per- formance in Table 1, our solvers use about 20x more MWPs than the original training set to train. Overall, results indicate that our knowledge dis- tillation approach achieves superior performance with much fewer parameters than the prior state-of- the-art LLMs, and outperforms the best fine-tuning and knowledge distillation baselines on all datasets.
Moreover, can clearly see that the improve- ment more noticeable in the OOD scenario. A possible reason for this could be that in the ID situ- ation, where the training and testing sets have some shared knowledge components, using random gen- eration for the source problems in the training set also helps to enhance the performance on the test- ing set. On the other side, in the OOD scenario, where there's large gap between the training and testing sets, our approach of creating tailored ex- ercises specifically targets the weak points of the student model, leading more effective boost in its accuracy.
4.5 Analysis on Exercise Book
We conducted experiment to demonstrate the effectiveness of our proposed exercise book by placing the original training set with the exercise book and changing the size m. As illustrated in Table 3, the performance of generating a same-size exercise book significantly better than that of us-

---

Case I: Equivalent problem generation
This case shows that LLM generales an equivalent problem using different key words. In school there NO girls and N1 boys. How many girls than boys does the school have?
Source
Problem
Answer
NO-N1
Generated
In a zoo there are NO lions and N1 tigers. How many more lions than tigers are there in the zoo?
Problem
Answer
NO
Case Sub-problem generation
This case demonstrates instance where sub-prohlem is derived from the source problem.
Source
In a school there are NO girls and N1 boys. N2 more girls joined the school. How many more girls than boys does the school have now?
Problem
Answer
NO N1
Generated
In school there NO girls and N1 boys. N2 more girls joined the school. How many girls there in total now?
Problem
Answer
N N2
Case 3: Incorrect format generation
This case shows a problem where the LLM generates a problem with incorrect formatting.
Source
Gwen's birthday she received NO dollars. She spent some money and NI dollars left. How much money she spend?
Problem
Ánswer
NO-NI N1
Generated
For Gwen's 's birthday she received NO dollars. She spent some money and now has Ni dollars left. How much money did she originally have?
Problem
Answer
NO N1 spent amount (Wrong format)
Table 4: Examples of problem generation by our method. Case and Case (colored in green) show successful instances where mathematically equivalent problem and sub-problem generated respectively. Case 3 (colored in red) illustrates a problem with incorrect formatting.
ing the training set itself for validation. Validating the training cannot fully reflect the learn- ing status of the student model, which may simply memorize the solutions of training problems and is not robust to slightly perturbed problems. This strongly confirms that our proposed exercise book approach can identify weaknesses of the student model. Furthermore, by increasing the exercise book size, more knowledge can be covered, lead- ing further improvement in accuracy. It is worth noting that the "Same Size Training Set" method has similar performance removal of problems with wrong formats and dupli- cated problems, as stated in Section 3.4. Therefore, the size of the exercise book under these two set- tings is empirically similar.
Performance Comparison of Training Methods
72.5
70.0
Performance
67.5
80.0
One-lime Bese
Progreesive Base
57.5
One-dime
Progressive Large
16000
2" due to the
Size of Training Set
Figure 3: Performance Comparison between one- one-time augmentation and progressive augmentation SVAMP under out-of- d-dittibbuuinn setting.
4.6 Analysis on Progressive Augmentation
To evaluate the efficacy of our proposed progressive and customized training framework, we compare it against the one-time data augmentation approach. Specifically, one-time data augmentation means that, we augment the size of the training set the beginning of the training process to be the same as the final size of the training set in our proposed framework and evaluate the performance of the stu-
dent MWP solver on SVAMP-OOD. Intuitively, the one-time augmentation will be better than progres- sive augmentation because it accesses the entirely augmented training set earlier. However, shown in Figure 3, the results indicate that our progres- sive training method outperforms the one-time data augmentation approach in terms of enhancing the performance of the student solver.

---

4.7 Case Study
Our current filtering process only eliminates prob- lems with incorrect formatting. Thus, there exists a significant opportunity for enhancing the effective- ness of our approach by incorporating mechanisms for evaluating and ensuring the quality and correct- ness of the generated exercises.
In Table 4, we present several example problems generated by our method, comprising both high- quality and problematic outputs. The first case demonstrates the LLM's ability produce a math- ematically equivalent problem using distinct key- words. The second successful instance yields a sub- problem derived from the source problem, which has been proven to facilitate a better understand- ing of mathematical word problems for models (Shridhar et al., 2022a). However, not all generated problems are idcal. As illustrated in the third exam- ple. our method occasionally generates MWPs with incorrect formatting, rendering them unsuitable for training our solver. As a result, we filter out such outputs to maintain the quality and accuracy of our training data, by checking if the generated answer is a valid mathematical equation.
References
Ghodai Abdelrahman, Qing Wang. and Bernardo Nunes. 2023. Knowledge tracing: A survey, ACM Comput- ing Surveys, 55(11):1-37.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. NerulPS, 33:1877-1901.
Wenhu Chen, Xueguang Ma, Xinyi Wang. and William W Cohen. 2022. Program thoughts
prompting: Disentangling computation from reason- ing for numerical reasoning tasks. arXiv preprint arXiv:2211. 12588.
5 Conclusion
In this work, we present a novel approach CEMAL to use large language models to facilitate knowl- edge distillation in math word problem solving. Our method first generates an exercise book to eval- uate student models and then provides additional training exercises that are customized to the learn- ing needs of the student model, thereby improving the student solver's ability to solve MWP. Our ex- tensive experimental results demonstrate that our proposed approach outperforms all fine-tuned and knowledge distillation baselines on all datasets, while achieving competitive performance against LLMs with significantly fewer parameters. Addi- tionally, explore different selection generation strategies, revealing that our proposed customized generation strategy is the most effective method, especially in the in-distribution setting. In our fu- ture work, we aim to extend this approach to other NLP tasks to evaluate its generalization capability,
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Albert T Corbett and John R Anderson. 1994. Knowl- edge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted inter- action, 2-223-77..
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. 2019, Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACI, pages 4171-4186
Shizhe Diao, Pengcheng Wang. Yong Lin, and Tong Zhang. 2023. Active prompting with chain-of- thought for large language models. arXiv preprint 2xiv:2332..22446
Charles R Fletcher. 1985. Understanding and solving arithmetic word problems: A computer simulation. Behavior Research Methods, Instruments, & Comput- ers, 17555565--571
Limitations
Peggy Grant and Dale Basye. 2014. Personalized learn- ing: A guide for engaging students with technology. International Society for Technology in Education.
Firstly, our approach necessitates meticulous prompt design to generate exercises, which in- evitably entails human intervention. This aspect could introduce potential bias or variability and may not scale efficiently.
John Hattie and Helen Timperley. 2007. The power of feedback. Review of educational research, 77(1):81- 112.
Secondly, we have not explicitly addressed the quality and correctness of the generated problems.
Mary Hegarty. Richard E Mayer, and Christopher A Monk. 1995. Comprehension of arithmetic word problems: A comparison of successful and unsuc- cessful problem solvers. Journal of educational psy- chology, 87)1:818
Codes and data are available at https: httts://ggttubb.cmm Caanwen-NLP/CENE

---

Namgyu Ho, Laura Schmid, and Sc- Young Yun. 2022. Large language models are reasoning teachers. arXiv preprint 100707
Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian. Baolin Peng, Yi Mao, et al. 2022a. Explanations from large language models make small reasoners better. arXiv preprint arXiv:2210.0672:
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 98yx1735- 1780.
Zhongli Li, Wenxuan Zhang, Chao Yan, Qingyu Zhou, Chao Li, Hongzhi Liu, and Yunbo Cao. 2022b. Seek- ing patterns, not just memorizing procedures: Con- trastive learning for solving math word problems. Findings of ACI.
Mohammad Javad Hosseini, Hannaneh Hajishirzi. Oren Etzioni, and Nate Kushman. 2014, Learning to solve arithmetic word problems with verb categorization. In EMNLP, pages 523-533.
Zhenwen Liang, Dian Yu, Xiaoman Pan, Wenlin Yao, Qingkai Zeng, Xiangliang Zhang, and Dong Yu. 2023. Mint: Boosting generalization in mathematical reasoning via multi-view fine-tuning. arXiv preprint arXiv:2307.07951.
Danqing Huang, Shuming Shi, Chin-Yew Lin, and Jian Yin. 2017. Learning fine-grained expressions solve math word problems. EMNLP, pages 805-814.
Shifeng Huang, Jiawei Wang, Jiao Xu, Da Cao, and Ming Yang. 2021. Recall and leamn: A memory- augmented solver for math word problems. In Find- ings of EMNLP, pages 786-796.
Zhenwen Liang, Jipeng Zhang. Lei Wang, Wei Qin. Yunshi Lan, Jie Shao, and Xiangliang Zhang. 2022a. Mwp-bert: Numeracy-augmeent
pre-training for math word problem solving. In NAACL, pages 997- 1009.
Zhanming Jic, Jierui Li, and Wei Lu. 2022. Learning to reason deductively: Math word problem solving as complex relation extraction. In ACL, pages 5944- 5955.
Zhenwen Liang, Jipeng Zhang, and Xiangliang Zhang, 2022b. Analogical math word problems solving with enhanced problem-solution association. EMNLP.
Diederik P Kingma and Jimmy Ba. 2015. Adam: method for stochastic optimization. ICLR.
Zhenwen Liang and Xiangliang Zhang. 2021. Solving math word problems with teacher supervision. In IJCAI pages 3522-3528.
Kenneth R Koedinger and Mitchell J Nathan. 2004. The real story behind story problems: Effects of represen- tations on quantitative reasoning. The journal of the learning sciences, 1322
Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2022. Wanli: Worker and ai collabora- tion for natural language inference dataset creation. Findings of EMNLP.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. NeurIPS.
Qianying Liu, Wenyu Guan, Sujian Li, Fei Cheng, Daisuke Kawahara, and Sadao Kurohashi. 2021. Roda: reverse operation based data augmentation for solving math word problems. IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, 30:1-11.
Rik Kedzio--Kd Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equa- tions. TACL, 597.
Qianying Liu, Wenyy Guan, Sujian Li, and Daisuke Kawahara. 2019a. Tree-structured decoding for solv- ing math word problems. In EMNLP, pages 2370- 2379.
Rik Mennell-Kdzzorski. Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. Mawps: A math word problem repository. In NAACL, pages 1152-1157.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.1 1692.
Nate Kushman, Yoav Artzi, Luke Zettlemoyer and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. In ACL, pages 271 1 281.
Ilya Loshchilov and Frank Hutter, 2018. Decoupled weight decay regularization. In ICLR.
Yihuai Lan, Lei Wang. Qiyuan Zhang, Yunshi Lan, Bing Tian Dai, Yan Wang, Dongxiang Zhang, and Ec- Peng Lim. 2022. Mwptoolkit: An open-souree frame- work for deep learning-based math word problem solvers. In AAAI, volume 36, pages 13 1318--13190.
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Erie Malmi, and Aliaksei Severyn. 2022. Teaching small language models to reason. arXiv preprint ar 22X
Jierui Li, Lei Wang, Jipeng Zhang, Yan Wang. Bing Tian Dai, and Dongxiang Zhang. 2019. Modeling intra- relation in math word problems with different fune- tional meat attentions. ACL, pages 6162- 6167.
Shen-Yun Yun Miao, Chao-Chun Liang, and Keh-Yit Reh-Yih Su. 2020. A diverse corpus for evaluating and developing english math word problem solvers. In ACL, pages 975-984.

---

Arindam Mitra and Chitta Baral. 2016. Learning to use formulas to solve simple arithmetic problems. In ACI, pages 2144-2153.
Yufei Wang. Can Xu, Qingfeng Sun. Huang Hu. Chongyang Tao, Xiubo Geng, and Daxin Jiang. 2022. Promda: Prompt-based data augmentation for low- resource nlu tasks. In ACL, pages 4242-4255.
Anirban Mukherjee and Utpal Garain. 2008. review of methods for automatic understanding of natural language mathematical problems. Artificial Intelli- gence Review, 29:93-122.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama. Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a. Emergent abilities large language models, Transactions on Machine Learning Research.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? In NAACL, pages 2080-2094,
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou, 20226, Chain of thought prompting elicits reasoning in large language models. NeurlPS.
Subhro Roy and Dan Roth. 2015. Solving general arith- metic word problems. In EMNLP, pages 1743-1752.
Subhro Roy. Tim Vieira, and Dan Roth. 2015. Rea- soming about quantities in natural language. TACL, 3:1-13.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang. Liwei Jiang. Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- els to commonsense models. In NAACL, pages 4602- 4625.
Zhihong Shao, Yeyun Gong, Yelong Shen, Min- lie Huang, Nan Duan, and Weizhu Chen. 2023. Synthetic prompting: Generating ofain-of-thought demonstrations for large language models. arXiv preprint ar 23:2222300.000 00..0
Zhipeng Xie and Shichao Sun. 2019. goal-driven tree-structured neural model for math word problems. In IJCAL, pages 5299-5305.
Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. 2021. Generate rank: A multi-task framework for math word prob- lems. In Findings of EMNLP, pages
Zhicheng Yang. Jinghui Qin, Jiaqi Chen, and Xiao- dan Liang. 2022. Unbiased math word problems benchmark for mitigating solving bias. Findings of NAACL.
Shumming Shi, Yuehui Wang, Chin- Lin, Xiaojiang Liu, and Yong Rui. 2015. Automatically solving number word problems by semantic parsing and soning. EMNLP, pages 1132-1142.
Jiacheng Ye, Jiahui Gao, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022a. Progen: Pro- gressive zero-shot dataset generation via in-context feedback. Findings of EMNLP.
Kumar Shridhar, Jakub Macina, Mennatallah El-Assady, Tanmay Sinha, Manu Kapur, and Mrinmaya Sachan. 2022a. Automatic generation of socratic subques- tions for teaching math word problems. In EMNLP.
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng. Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022b. Zerogen: Efficient zero-shot learning via dataset generation. EMNLP.
Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2022b. Distilling multi-step reasoning capa- bilities of large language models into smaller mod- els via semantic decompositions. arXiv preprint arXiv:2212.00/99
Jipeng Zhang, Lei Wang, Roy Ka- Ke-Wei Lee, Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim. 2020. Graph-to- tree learning for solving math word problems. In ACL, pages 3928-3937.
Chujie Zheng. Sahand Sabour, Jiaxin Wen, and Minlie Huang. 2022. Augesc: Large-scale data augmentation for emotional support conversation with pre-trained language models. arXiv preprint arXiv:2202.13047
Ben Wang and Aran Komatsuzaki. 2021. Gpt- j-6b: A billion parameter autoregressive lan- guage model. setpps//gitthen wtw.:tuwb.com/iigooooolo.oo mesh-trrassFommer-1
Lei Wang, Yan Wang. Deng Cai, Dongxiang Zhang, and Xiaojiang Liu. 2018. Translating a math word problem to expression tree. In T.,LPP, pages 1064- 1069.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quee Le, and Ed Chi. 2023. Least- to-most prompting enables complex reasoning in large language models. ICLR.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Fd Chi, and Denny Zhou. 2023. Self-consisenccy im- proves chain of thought reasoning language mod- els. ICLR.
Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In EMNLP, pages 845-854.

---

Source Problem: In school there Ne girls and N1 boys. N2 more girls joined the school. How many more girls than boys does the school have now?
Source Problem
Answer: No + N2 N1
Generate a math word problem and its answer with exactly same format as the previous one without explanation, using NO, N1, ... to represent numbers without other number representations.
Instruction
Variant Problem:
Figure 4: The zero-shot prompts generate solution-analogy MWP pairs.
A
Appendix
A.1 Prompts
We have two different prompting strategies, i.e.. few-shot prompting and zero-shot prompting. The former gives the LLM examples of how to generate a variant problem given a souree problem, as shown in Figure 5. The full list of exemplars can be found in Table 5. The latter only use instructions to guide LLM in generating similar MWP, shown in Figure The examples of generated problems can be found in Table 4. Empirically, we use the ChatGPT gpt-3.5-turbo API with temperature of 0.9 to perform generation.

---

Generate variant for math word problems, the answer for variant should be correct and different from the source, with a exactly same format as the source, contain not only operands but also operators, do not contain mod operator, and using Ne, NI, to represent numbers without other number representations.
Instruction
Source Problem: After storm No trees in Haley's garden perished. Originally, if she had N1 trees, how many trees were unaffected compared to the ones that succumbed to the storm?
Answer: N1
Ne
Few-shot Exemplars
Variant Problem: After typhoon NO trees Haley's 's backyard died. If she had grown N1 trees initially, how many trees does she have remaining?
Answer: Ne
Source Problem: school there are girls N1 boys. N2 more girls joined the school. How many more girls than boys does the school have now?
Source Problem
Answer: NB N1
Variant Problem:
Figure 5: The few-shot prompts to generate problcm-analogy MWP pairs.
Problem
Answer
After a storm NO trees in Haley's garden perished. Originally, if she had NI trees, how many trees were unaffected compared to the ones that succumbed to the storm?
Source Problem:
NI NO NO
Example
Variant Problem: After typhoon NO trees in Haley's back- yard died. If she had grown NI trees initially, how many trees does she have remaining?
N1 NO
Source Problem: An industrial machine made NO shirts yes- terday and NI shirts today. It can make N2 shirts minute. How many minutes did the machine work yesterday? Variant Problem: An industrial machine made NO shirts yesterday and N1 shirts today. It can make N2 shirts minute. How many minutes did the machine work in all? Source Problem: NO red peaches, NI yellow peaches, and N2 green peaches are in the basket. How many peaches are in the basket?
NO / N2
Example 2
(NO
NO + NI + +N2
Example
Variant Problem: NO red peaches, NI yellow peaches, and N2 green peaches are in the basket. How many more red peaches than yellow peaches are in the basket?
NO NI
Table 5: The Full List of Few-Shot Examples


Reviews:
[{"review_id": "ka7KV1Lmyi", "reviewer": "Reviewer_Stk4", "summary": "", "questions": "(A) What might be potential strategies to enhance the quality and correctness of the generated problems? It would be good to further discuss this.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and insightful question.\n\nThere are two potential strategies to enhance the quality and correctness of the generated problems :\n\n1） **Implementation of Verifiers:** Building verifiers, as suggested in [1][2], could serve to score the exercises generated by our LLM teacher. A set threshold could then be employed to select high-quality problems.\n\n2） **Utilization of Well-Trained LLMs:** Models like ChatGPT could be leveraged to evaluate and potentially enhance the quality and correctness of the generated exercises.\n\nBoth approaches present intriguing possibilities, and we intend to explore them further in our future research. Thanks again for reviewing our paper and we look forward to your further feedback.\n\n[1] Training Verifiers to Solve Math Word Problems. Cobbe et al. 2021\n\n[2] Let's Verify Step by Step. Lightman et al. 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposed a novel method to distill a large language model's capability of solving math word problems into smaller but more efficient student models, and the method, to certain degree, is novel. Particularly, the idea of incorporating the principles of knowledge tracing and personalised learning to improve the knowledge distillation process is innovative.\n2. The experimental evaluation on three datasets is very comprehensive.", "reasons_to_reject": "None", "questions_to_authors": "(A) What might be potential strategies to enhance the quality and correctness of the generated problems? It would be good to further discuss this.", "paper_topic_and_main_contributions": "This paper proposed a novel method to distill a large language model's capability of solving math word problems into smaller but more efficient student models. The key contributions of this paper include: (1) the proposed method named CEMAL is based on the novel idea of improving a student model's performance by characterising its learning needs (or just say \"knowledge gap\") and subsequently providing it with tailored exercises to improve its learning performance; and (2) the developed method incorporates the core ideas of two longstanding tasks in education, namely knowledge tracing and personalised learning, which can shed light on how knowledge distillation can be performed to improve an AI model's capability in solving educational tasks. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}, {"review_id": "YlNe7HfbgX", "reviewer": "Reviewer_xeUf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and for providing constructive feedback on our manuscript. We sincerely appreciate your time and effort in evaluating our work. We hope that the response below can address your concerns. \n\n\n**Clarity of Methods and Evaluations**\nWe apologize for the difficulties you have faced while trying to follow the methods and evaluations in our paper. We are committed to substantially revising the manuscript to make it clearer. The revision will focus on: :\n\n**Figure 2 and Algorithm 1 for better introducing our solution:** We will revise Figure 2 to make it more straightforward for illustrating the workflow, eliminating those distracting elements. We will also revise Algorithm 1 and synchronize its description with Figure 2 for better coherence.\n\n**Model Names:** To remove any confusion related to the naming of the models used in our study, we will standardize the terminology. Throughout the paper, Large Language Models (LLMs) will be referred to as \"Teacher Models,\" and our trained solvers will be called \"Student Models.\" We will also attach a section to introduce them by detailing the API names of teacher models and network structures of student models.\n\n**SVAMP Dataset Description:** We understand that our original presentation of the SVAMP dataset was confusing. However, the SVAMP dataset is originally a test-only dataset without any training data. To utilize this dataset for evaluation, a commonly accepted way is to use MAWPS and ASDiv-a as the training sets, where our paper follows this setting and we were trying to explain this choice. In the revised version, we will clarify that SVAMP is a test-only dataset and detail our intentions.\n\n**Questions of Novelty and Relations to Prior Work**\nWe appreciate this comment. Our motivation is located in the Line 86-100 in our submission. In our revision, we will provide a clearer explanation of how our approach diverges from existing work. Specifically, our CEMAL methodology offers a new perspective on knowledge distillation in machine learning by focusing on customized exercise generation, based on the student models' current learning state. This enables more effective and adaptive learning, addressing limitations in prior approaches that rely heavily on generating more high-quality explanations to existing datasets.\n\n**Significance Study**\nThanks for the valuable comment! We acknowledge the importance of reporting statistically sound results. However, it is challenging to conduct a direct statistical significance analysis. Specifically, for the few-shot CoT methods, they utilize models like code-Davinci-002 and PaLM-540B, which are not publicly accessible. Similarly, the code and augmented datasets for the knowledge distillation baselines are not available, making them difficult to reproduce. Despite these challenges, we can confirm that backbones models with our proposed CEMAL outperforms the models without CEMAL, providing a strong indication of its effectiveness. Moreover, we calculated the standard deviation of the results in Table 1, as shown below, which can confirm that our method is stable in its performance.\n\n| Method                    | MAWPS                  | ASDiv-a               | SVAMP (ID)            | SVAMP (OOD)           |\n|---------------------------|------------------------|-----------------------|-----------------------|-----------------------|\n| LSTM + CEMAL              | 91.8 ± 0.21             | 86.7 ± 0.30            | 67.3 ± 0.15           | 53.3 ± 0.52            |\n| RoBERTa-base + CEMAL      | 93.7 ± 0.19             | 90.7 ± 0.21            | 81.6 ± 0.13            | 68.6 ± 0.38           |\n\n**Reproducibility Concerns**\nWe noted the reviewer's low score on the reproducibility of our work, and we wish to address this concern directly. Alongside with our submission, we have provided the necessary Materials like code and data and we also promised to release them after the notifications come out. We are fully committed to making our work easily reproducible by other researchers in the field.\n\nThank you once again for taking the time to review our paper and provide valuable feedback. We're committed to improving the manuscript and are more than willing to address any additional concerns you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper focuses on an important unsolved problem, which is to automatically solve math word problems. The authors thoroughly explore and evaluate several datasets and modeling approaches. Their exploration of targeted vs random and iterative vs one time augmentation are both quite intriguing. Overall, I think this paper would merit publication, and would be of interest to the ACL community, but it requires significant revision to improve clarity.", "reasons_to_reject": "1) The paper was extremely hard to follow. I read it multiple times and still had trouble following the exact experimental procedures and evaluations that the authors conducted. \n2) Relatedly, it was hard to discern what was novel in the paper and what had already been tried by others. \n3) Since the improvement in numbers is not large (in most cases, just a couple of points), it is hard to tell if this improvement is statistically significant and if it translates to qualitative improvements in performance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes an approach to distill math word problem solving capabilities from LLMs to smaller models. Their approach involves data augmentation (random and targeted), iterative vs one-time augmentation, testing the model on various dataset in and out of distribution, and trying out different base models.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I recommend substantially revising the paper to make it more clear:\n1) Figure 2 is meant to carry a lot of the load in providing an overview of the workflow, but it is very hard to read. Can you make it more clear?\n2) It was hard to keep track of the numerous models used for different procedural steps (RoBERTa models, GPT models, LSTM), it'd be good to clarify what is used when and give more descriptive names to each type of approaches/models so that the reader can keep track.\n3) The methods and results are sometimes mixed together in confusing ways. For example, in the Datasets section under SVAMP, you also describe methods you used in combination with the data and also include other datasets in the description, which makes it very confusing. It'd be useful to cleanly separate descriptions of datasets, then how they were processed, and then used for model development and evaluation."}}, {"review_id": "Psu6030SXe", "reviewer": "Reviewer_weic", "summary": "", "questions": "B Why CEMAL significantly outperforms prior Knowledge Distillation in out-of-distribution testset  (20.7 vs 76.4), but only a little in in-distribution testset (94.5 vs 94.7)? Intuitively speaking, out-of-distribution should be much more complex than in-distribution and thus harder to improve, for example, your method outperforms Few-shot CoT on in-distribution, but not outperforms Few-shot CoT on in-distribution.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to A:**\nThanks for the valuable comment! We acknowledge the importance of reporting statistically sound results. However, it is challenging to conduct a direct statistical significance analysis. Specifically, for the few-shot CoT methods, they utilize models like code-Davinci-002 and PaLM-540B, which are not publicly accessible. Similarly, the code and augmented datasets for the knowledge distillation baselines are not available, making them difficult to reproduce. Despite these challenges, we can confirm that backbones models with our proposed CEMAL outperforms the models without CEMAL, providing a strong indication of its effectiveness. Moreover, we calculated the standard deviation of the results in Table 1, as shown below, which can confirm that our method is stable in its performance.\n\n| Method                    | MAWPS                  | ASDiv-a               | SVAMP (ID)            | SVAMP (OOD)           |\n|---------------------------|------------------------|-----------------------|-----------------------|-----------------------|\n| LSTM + CEMAL              | 91.8 ± 0.21             | 86.7 ± 0.30            | 67.3 ± 0.15           | 53.3 ± 0.52            |\n| RoBERTa-base + CEMAL      | 93.7 ± 0.19             | 90.7 ± 0.21            | 81.6 ± 0.13            | 68.6 ± 0.38           |\n\n**Response to B:**\nIt is true that in-distribution (ID) datasets are usually easier than out-of-distribution (OOD) datasets. However, our method aims to further improve the performance via customized exercise generation. In this case, for ID datasets, the knowledge components between training and testing have larger overlap, making it more challenging to generate effective customized exercises that would lead to performance improvements. In contrast, for OOD datasets, the student model cannot learn very well on the original training set, but our method can more effectively identify the model's weaknesses, thus improving its performance via customized exercises. We appreciate your insightful question and will include this clarification in our revised paper.\n\nThank you for your time for reviewing our paper. We look forward to your further feedback and are happy to address any concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A The proposed CEMALoutperforms fine-tuned and knowledge distillation baselines and achieves competitive results with LLMs. (Table 1)\n\nB Using hard examples to prompt LLM teacher to generate exercises outperforms easy example on the student math accuracy, where hard examples are the ones that SLM student fails. (Table 2) This matches with the intuition proposed in the intro.\n", "reasons_to_reject": "A Is the results in Table 1 and Table 2 statistically stable? Could you report standard deviation?", "questions_to_authors": "B Why CEMAL significantly outperforms prior Knowledge Distillation in out-of-distribution testset  (20.7 vs 76.4), but only a little in in-distribution testset (94.5 vs 94.7)? Intuitively speaking, out-of-distribution should be much more complex than in-distribution and thus harder to improve, for example, your method outperforms Few-shot CoT on in-distribution, but not outperforms Few-shot CoT on in-distribution.", "paper_topic_and_main_contributions": "The authors demonstrate a novel approach for distilling math word problem-solving capabilities from large language models (LLMs) into smaller, more efficient student models. Even though there are many existing works on distilling LLMs using the explanation from LLMs, smaller models tend to misunderstand CoT explanations. Thus the authors look from a different perspective. They used LLMs as math tutors that generate customized exercises for the student models according to their learning status and feedback (CEMAL), which is inspired from educational science principles, such as knowledge tracing and personalized learning. The paper evaluates its approach on multiple math word problem datasets, and shows that it outperforms fine-tuned and knowledge distillation baselines, and achieves competitive results with LLMs, especially in the out-of-distribution setting. \n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. Interpretable Math Word Problem Solution Generation Via Step-by-step Planning ACL 2023 https://aclanthology.org/2023.acl-long.379.pdf \n2. REACT: SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS ICLR 2023 https://openreview.net/pdf?id=WE_vluYUL-X\n3. Practice Makes a Solver Perfect: Data Augmentation for Math Word Problem Solvers https://aclanthology.org/2022.naacl-main.310/\n\nat line 182"}}]

Author rebuttal:

Meta-review: The manuscript presents a contribution in the field of knowledge distillation, proposing the CEMAL method, which distills math word problem-solving capabilities from large language models (LLMs) into smaller student models, which has proven previously to have problems with CoT. This approach, inspired by educational science principles like knowledge tracing and personalized learning, addresses common limitations of existing methods and outperforms them in both in-distribution and out-of-distribution settings. The manuscript provide a thorough experimental evaluation. The main concerns is the presentation of findings and highlighting exact contributions in relation to existing techniques. We encourage authors, as stated in the rebuttal to address these concerns to make their manuscript more accessible to the community.
Decision: Accept-Main