Rationales for Answers to Simple Math Word Problems Confuse Large Language Models (2024)

Abstract: Recently, large language models (LLMs) have demonstrated breakthrough mathematical problem-solving capabilities in grade school math word problems (MWP). For example, on the MWP benchmark GSM8K, the accuracy of GPT-3.5-Turbo and MetaMath-70B reaches 80.80% and 82.30%, respectively. One question arises, does it mean that LLMs have truly mastered related mathematical problemsolving abilities? In this paper, by presenting two types of benchmarks, where MCGSM8K aims at selecting one correct solution from four solutions, while GSM8K-Judgement judges whether a solution to a given question is true or false, we demonstrate that the ability of most LLMs to evaluate the mathematical reasoning process of MWP is far from sufficient. To compensate for this issue, we propose hybrid supervised fine-tuning data from the training data of GSM8K, MCGSM8K, and GSM8KJudgement, which significantly improves performance on the proposed reasoning process evaluation benchmarks. For example, finetuning improves the performance of LLaMA-213B from 33.51% to 70.89% on MCGSM8K. In conclusion, we experimentally demonstrate that most LLMs have limited ability to evaluate the mathematical reasoning process of MWP, which can be enhanced through fine-tuning.

heading: 1 Introduction
text: It is reported that general close-source large language models (LLMs) have demonstrated promising performance on several mathematical word problems (MWP) benchmarks, e.g., GPT-4 (OpenAI, 2023) and GPT-3.5-Turbo (OpenAI, 2022) achieving the accuracy of 92.00% and 80.80% on grade school MWP benchmark GSM8K (Cobbe et al., 2021a), respectively. With the development of prompt-based methods (Fu et al., 2023; Wang et al., 2023a) and finetuning-based methods (Yu et al., 2023; Yue et al., 2023; Yuan
∗ Corresponding author.
et al., 2023), mathematical specialized LLMs tuned on specific tasks also exhibit competitive performance. For example, MetaMath-70B (Yu et al., 2023), WizardMath-70B (Luo et al., 2023), and MAmmoTH-70B (Yue et al., 2023) achieves 82.30%, 81.60%, and 76.90% on GSM8K, respectively. Now, one question arises, does the excellent performance demonstrate that these LLMs truly master related mathematical problem-solving abilities, such as the ability to evaluate the mathematical reasoning process of MWP?
Intuitively, picking one correct solution from possible solutions is easy for humans, as it just requires evaluating the correctness of the reasoning process. In comparison, reasoning the answer based on the open-formed question is difficult, which requires analyzing the problem, step-bystep reasoning, and deriving the final result (Cobbe et al., 2021b). Building upon this premise, we design a simple mathematical reasoning processes evaluation benchmark, MCGSM8K 1 aiming at choosing one correct solution from four options, as shown in Figure 2. Then, we utilize a few-shot (Chen et al., 2022b; Min et al., 2022) setup to test the performance of typical general open-source models LLaMA-2, general closed-source models GPT-3.5-Turbo and GPT-4, as well as mathematical specialized models MetaMath and MAmmoTH on it. However, our experimental results in Figure 1 and Table 1 reveal that most LLMs lag far behind on MCGSM8K. For example, the accuracy of LLaMA-2-70B, MetaMath-70B, and GPT3.5-Turbo drop from 56.80% to 38.29%, 82.30% to 34.87%, and 80.80% to 40.56%, respectively. Specifically, each solution of MCGSM8K contains both the final answer and a step-by-step reasoning process (rationale). We collect incorrect solutions by regenerating solutions for each question in the test set of GSM8K. To keep the quality and di-
1MCGSM8K is publicly available at https://github. com/SCUZPP/MCGSM8K.git
8853
Average testing accuracy
versity of MCGSM8K, we use multiple advanced LLMs, e.g., Qwen (Bai et al., 2023), LLaMA-2 (Touvron et al., 2023b), MetaMath (Yu et al., 2023), and WizardLM (Xu et al., 2023), through few-shot Chain-of-thought (CoT) (Wei et al., 2022) prompting to generate incorrect solutions.
To comprehensively investigate the performance of LLMs on MCGSM8K, we progressively conduct the following experiments with a few-shot setting on three well-designed benchmarks as shown in Figure 2. First of all, the main reason for the poor performance of LLMs on MCGSM8K might include a) the model’s inability to solve multiplechoice format questions, and b) the model’s inability to evaluate the reasoning process. To verify, we propose a conventional multiple-choice-question benchmark MCGSM8K-No-Rationale by removing the rationale and leaving only the final answer for each option from MCGSM8K. The average accuracy of all tested models on MCGSM8K-NoRationale (63.96%) is 25.50% higher than that on MCGSM8K (41.59%) and close to that on GSM8K (68.98%). The result reveals that the poor performance might be due to the model has difficulty in evaluating the reasoning process instead of the answer directly. Furthermore, we analyze the performance of LLMs by reducing the difficulty of solving the problem in MCGSM8K to half. Specifically, we remove any two incorrect options for each question in MCGSM8K, resulting in MCGSM8K-2Options aiming at selecting one correct solution from two. However, the average accuracy of all tested models on the twochoice-question benchmark MCGSM8K-2Options is merely 1.17% higher than that on the openformed-question benchmark MCGSM8K (70.15% vs. 68.98%). MetaMath-70B and LLaMA-270B achieve an accuracy of 66.94% and 64.22%, which is merely 16.94% and 14.22% higher than the random-chance accuracy of 50%, respectively. This verifies the ability of most LLMs to evaluate the mathematical reasoning process is insufficient. Are LLMs insufficient to evaluate the correct solution or the incorrect solution? To figure this out, we propose a true-or-false-question benchmark GSM8K-Judgement to directly judge the correctness of the solution to a given question. In general, it is easier for humans to identify incorrect solutions than correct solutions, as the former only requires identifying a certain incorrect step, while the latter must ensure the correctness of all steps. Experimental results on general open-source models show that the average True Negative rate (TNR) is significantly better than the average True Positive rate (TPR), which is consistent with human behavior. However, when it comes to mathemat-
ical specialized models and GPT-3.5-Turbo, the situation is completely reversed.
Through the above experimental analysis, it can be concluded that most LLMs have a poor ability to evaluate the mathematical reasoning process of MWP. Mathematical specialized models are mainly fine-tuned on abundant correct solutions, which greatly improves the ability to identify correct solutions while causing catastrophic forgetting in identifying incorrect solutions. In addition, we hypothesize that most LLMs only catch spurious signals in specific datasets resulting in “solving” the datasets while not mastering abilities related to mathematical problem-solving.
In this paper, we try to compensate for these shortcomings by finetuning models on hybrid training samples from GSM8K, MCGSM8K, and GSM8K-Judgement. This contributes to enhancing the generalization ability to solve mathematical problems, mastering new data distribution on the MCGSM8K and GSM8K-Judgement, identifying correct solutions, as well as learning to analyze and evaluate the reasoning process. Specifically, we use LLaMA-2-13B as the base model. After fine-tuning, we observe a substantial improvement in accuracy with an increase of +37.38% on MCGSM8K, +41.24% on TPR, +8.87% on TNR, +7.43% on MCGSM8K-No-Rationale, and +16.62% on GSM8K. The result demonstrates that fine-tuning can greatly improve the mathematical reasoning process evaluation ability of LLMs.
Our main contributions can be summarised as follows:
• To explore whether LLMs have mastered the ability to evaluate the mathematical reasoning process of MWP, we carefully create two kinds of benchmarks. The first category is multiple-choice questions, including MCGSM8K aiming at choosing one correct solution from four options, MCGSM8K2Options containing only two options, and MCGSM8K-No-Rationale with only the final answer in each option. The second type is true or false questions, including GSM8KJudgement to judge the correctness of the solution to a given question.
• We conduct experiments with typical general open-source models, general closed-source models, and mathematical specialized models on the four benchmarks. The experimental
results reveal that existing LLMs except GPT4 have a poor ability to evaluate the mathematical reasoning process of MWP. Meanwhile, fine-tuning with only correct solutions improves the performance in evaluating correct solutions, but leads to a huge performance drop in evaluating incorrect solutions. Furthermore, we experimentally demonstrate that these drawbacks of LLMs can be alleviated to a certain extent through fine-tuning.

heading: 2 Related Work
text: Large Language Models. LLMs with billions of parameters trained on extensive large-scale corpora of textual data have led to massive changes in the field of AI. OpenAI’s GPT series (Brown et al., 2020; OpenAI, 2023), which is the most representative general closed-source LLM, opens the era of pre-trained LLMs, where a large number of prominent instances are launched one after another involving Anthropic’s Claude 2 (Bai et al., 2022), Google’s PaLM (Chowdhery et al., 2023; Anil et al., 2023), DeepMind’s Chinchilla (Hoffmann et al., 2022), and Gopher (Rae et al., 2021). Subsequently, numerous general open-source LLMs have been released, whose code and weight parameters are open to the public. Typical examples include LLaMA (Touvron et al., 2023a,b), GLM130B (Zeng et al., 2023), OPT (Zhang et al., 2022), Falcon (Penedo et al., 2023), and so on. Although general closed-source LLMs, e.g., GPT-3.5, GPT-4, and PaLM-2, have achieved considerable advancements in several MVP tasks such as GSM8K and NumGLUE (Mishra et al., 2022), the performance of general open-source LLMs is far from satisfactory.
Large Language Models for Mathematical Reasoning. Chain-of-thought (CoT) (Wei et al., 2022) reasoning by designing better prompts has been proposed to generate step-by-step solutions leading to improved performance in complex reasoning. To improve the mathematical reasoning capabilities of general open-source LLMs, existing methods focus on CoT prompting for augmenting fine-tuning data. WizardMath (Luo et al., 2023) utilizes few-shot CoT prompting to re-generate solutions for GSM8K and MATH (Hendrycks et al., 2021), then uses this data to construct SFT data. MetaMath (Yu et al., 2023) aims to improve finetuning-based methods by answer augmentation and available mathematical questions bootstrap to
construct SFT data. RFT (Yuan et al., 2023) improves mathematical reasoning performance by applying CoT prompts and Rejection Sampling (RS) on SFT models to construct augmented solutions. MAmmoTH (Yue et al., 2023) utilizes a unique hybrid of CoT and program-of-thought (PoT) (Chen et al., 2022a) rationales to construct augmented solutions for improving mathematical problem-solving ability. As a result, these mathematical specialized models have surpassed previous general open-source LLMs by a significant margin in mathematical problem-solving.
Large Language Models for Mathematical Reasoning Process Evaluation. There are different ways to investigate the mathematical reasoning process, e.g., scoring each step of the reasoning process (Lightman et al., 2023), examining and analyzing each step, as well as modifying the reasoning process based on analyses (An et al., 2023). However, the above methods are mainly used to improve reasoning capabilities. Lightman et al. (2023) demonstrates that step-by-step verifying for the reasoning process of LLMs can lead to bet-
ter model performance in mathematical problemsolving. An et al. (2023) fine-tunes LLMs on mistake-correction data pairs generated by GPT4 to improve their reasoning capabilities in solving math problems. Existing research rarely explores the mathematical reasoning process evaluation ability of LLMs. We aim to design two types of benchmarks to test the ability of LLMs to evaluate the mathematical reasoning process of MWP. To achieve this goal, we design two direct and simple benchmarks, i.e., choosing the correct reasoning process from four candidates and judging whether a given reasoning process is correct or not. Please note that all existing mathematical reasoning benchmarks can be used to construct reasoning process evaluation benchmarks by our proposed method.
Similar to our work, there are also studies focusing on the counter-intuitive behaviors of LLMs. Berglund et al. (2023) studies the Reversal Curse of LLMs, i.e., LLMs exhibit a basic failure in deducting from “A is B” to the reverse direction “B is A”. Pezeshkpour and Hruschka (2023) aims to investigate the sensitivity of LLMs against options order
of multiple-choice questions and recommends two approaches to calibrate LLMs’ predictions including majority vote and multiple evidence calibration (MEC).

heading: 3 MCGSM8K
text: In this section, we describe in detail the process of constructing the benchmark MCGSM8K, which consists of multiple-choice questions with each question from the original test set of GSM8K and each option containing a solution. Our primary objective is to guarantee the reliability of MCGSM8K, ensuring that correct options are indeed correct and incorrect options are demonstrably wrong. To achieve this, we derive correct options directly from GSM8K, incorporating both the validated answer and its accompanying rationale, which are manually annotated to ensure accuracy. Conversely, incorrect options are generated by LLMs, comprising wrong answers distinct from the ground-truth answer, each accompanied by a plausible rationale. On the premise that the reliability of the dataset is guaranteed, we further enhance the challenge and difficulty of the dataset. Thus, we 1) use multiple advanced open-source LLMs to generate numerous candidate solutions with wrong answers, 2) maintain the diversity of incorrect option differences based on Rouge-L (Lin, 2005) scores and k-means cluster (Wong, 1979), and 3) ensure the confusability of the incorrect options with the correct option based on similarity ranking.

heading: 3.1 Generation of Incorrect Solutions
text: Given a question qi, we utilize 8-shot CoT prompting to re-generate N solutions {(rji , a j i ) : j = 1, ..., N} by advanced open-sourced models including Qwen (Bai et al., 2023), LLaMA-2 (Touvron et al., 2023b), MetaMath (Yu et al., 2023), and WizardLM (Xu et al., 2023), with sizes of 13B and 70B. We use each model to generate 50 candidate solutions. The principles for selecting these specific models are detailed in Appendix C. Specifically, a question qi is appended to a few demonstrations, then fed to an LLM for generating its answer aji along with rationale rji step-by-step. In the generation process, we follow Wang et al. (2023b) to adopt temperature sampling and set the temperature as 0.7. Then, find out those with a wrong answer according to the ground-truth answer to construct
the incorrect solution set Setaug.
Setaug ={(aji , r j i ) :
aji ̸= a∗i ; i = 1, ...,K; j = 1, ...N}, (1)
where K is the total number of questions, and a∗i is the ground-truth answer for question qi.

heading: 3.2 Diversity and Quality
text: To enlarge diversity, an incorrect solution to qi is added to Setaug only when its ROUGE-L similarity with any existing solutions to qi in Setaug is less than 0.9, to effectively reconcile the retention quantity with the desired diversity of solutions. We also remove the calculation annotations proposed in the original answer rationale of GSM8K from each solution to facilitate formatting consistency. Invalid solutions are identified and filtered out based on heuristics, e.g., too-long or too-short solutions, solutions contain codes, and solutions do not end in the specified format. Finally, the candidate solutions to qi are clustered into three categories using k-means clustering to ensure the diversity of solutions among each category. More details on data quality and diversity is provided in Appendix D.

heading: 3.3 Construction of Dataset
text: We have designed three different selection ways to avoid a single selection method causing a similar distribution or text of the three incorrect options. The ablation study on data construction is analyzed in Appendix B. One way is picking the one with the highest ROUGE-L similarity score to the correct option, which can ensure the confusability of the incorrect options. Another way is picking out the one with the lowest Perplexity (PPL) scored by language model WizardLM-70B (Xu et al., 2023). The lower the PPL, the solution is more natural and more consistent with the model’s generation preferences. The last one is the random selection to keep diversity. For a question qi, we select one solution from each of the three clusters to construct three incorrect options. At every election, we choose one way in turn. Then, the three incorrect options are combined with the correct one in a randomly shuffled order. Finally, we combine the question, options, and the correct option label into a multiplechoice question.

heading: 4 Experiments
text: In this section, the core of our analysis is how LLMs perform on the proposed benchmarks, what
GSM8K-Judgement Model GSM8K MCGSM8K MCGSM8K -No-Rationale MCGSM8K -2Options TPR TNR
factors are associated with the model performance, and whether the performance can be further improved through supervised fine-tuning, thus answering the question: do LLMs master the ability to evaluate the mathematical reasoning process of MWP?

heading: 4.1 Can LLMs Solve Simple Mathematical Reasoning Process Evaluation of MWP?
text: To verify this issue, we propose a mathematical reasoning process evaluation benchmark MCGSM8K (Figure 2) consisting of 1,319 samples aiming at choosing the correct one from four solutions. We utilize the settings of few-shot in-context learning and CoT prompting, which is shown in Appendix E Figure 5.

heading: 4.1.1 Models
text: We evaluate the testing accuracy on several representative models – (i) general closed-source models GPT-42 (OpenAI, 2023) and GPT-3.5-Turbo (OpenAI, 2022), (ii) general open-source models: the current state-of-the-art LLaMA-2 (Touvron et al., 2023b) with two different parameter sizes of 13B and 70B, and (iii) mathematical specialized mod-
2We use gpt4-1106-preview.
els: MetaMath (Yu et al., 2023) and MAmmoTH (Yue et al., 2023) in sizes 13B and 70B, which specifically tune LLaMA-2 on mathematical reasoning datasets. MetaMath is tuned on mathematical reasoning datasets collected by mathematical questions bootstrap and answering augmentation. MAmmoTH is trained on an instruction-tuning dataset compiled from 13 math datasets with a unique hybrid of CoT and PoT rationales. In all evaluation experiments, we set a temperature of zero for open-source models and mathematical specialized models following previous work (Yu et al., 2023; Yue et al., 2023), and a temperature of 0.2 for general closed-source models to generate quality answers.

heading: 4.1.2 Results
text: In Table 1, we cite the metrics of all tested models on GSM8K from the paper of MetaMath (Yu et al., 2023) and MAmmoTH (Yue et al., 2023). We can see that general closed-source models and mathematical specialized models have shown promising performance in GSM8K. For example, the accuracy of GPT-3.5-Turbo and MetaMath-70B all exceeds 80% from the third column in Table 1. These results exhibit the strong ability of most LLMs to
solve grade school math word problems. However, all models except GPT-4 achieve poor performance below 41% on the simple MCGSM8K compared with their performance on GSM8K. The accuracy of LLaMA-2-70B, MetaMath-70B, and GPT-3.5-Turbo drop from 56.80% to 38.29%, 82.30% to 34.87%, and 80.80% to 40.56%, respectively. To maximize accuracy, we test model accuracy with and without CoT prompting. As illustrated in the second column of Table 4 in Appendix A, the average performance gap between the test accuracy with and without CoT prompting is tiny, which is within 5%, exhibiting the incompetence of most LLMs in solving problems from MCGSM8K. We have tried different CoT prompting shown in Appendix Table 3. Specifically, we prompt the model to describe the task and explain the answer. The performance improvement brought by different prompts is less than 3%, and the best result so far is still much lower than that on GSM8K.
Point 1: Although most LLMs can solve MWP to some degree, they have difficulty in evaluating the reasoning process of MWP.

heading: 4.2 Can LLMs Solve MWP in the Form of Multiple-choice Questions?
text: In this subsection, we conduct the second experiment to investigate whether LLMs are incapable of solving multiple-choice format questions or evaluating the reasoning process. Specifically, for each option of MCGSM8K, we remove the reasoning process (rationale) and leave only the final answer to construct MCGSM8K-No-Rationale (Figure 2) consisting of 1319 samples. The model setting is the same as that in subsection 4.1.1. We report the testing accuracy under the settings of 5-shot in-context learning and CoT prompting, as shown in Appendix E Figure 5.
The average CoT performance of all tested models on MCGSM8K-No-Rationale is significantly higher than that on MCGSM8K (63.96% vs. 41.59%), and close to that on GSM8K (63.96% vs. 68.98%). From the paper of MAmmoTH (Yue et al., 2023), GPT-4 and MAmmoTH-70B achieve an accuracy of 72.60% and 65.00%, on the AQuA (Ling et al., 2017) dataset consisting of multiplechoice algebraic word problems, respectively. The results exhibit the ability of most LLMs to solve MWP in the form of multiple-choice questions. In addition, CoT prompting brings significant performance gains, e.g., the performance improvement of LLaMA-2-70B, and MetaMath-70B is 20.85%,
and 38.74%, respectively, as shown in the third column of Table 4 in Appendix A.
Point 2: Most LLMs are capable of solving MWP in the form of multiple-choice questions.

heading: 4.3 Can Reducing Options on the Problem to
text: Be Solved in MCGSM8K Improve Model Performance?
For each sample in MCGSM8K, we remove any two incorrect options from the four options, leaving only one correct and one incorrect option, resulting in MCGSM8K-2Options (Figure 2) consisting of 1319 samples. The model setup is the same as that in subsection 4.1.1. and we use 8-shot in-context learning without CoT prompting.
As illustrated in Table 1, the average accuracy of all tested models on the two-choice-question benchmark MCGSM8K-2Options is merely 1.17% higher than that on the open-formed-question benchmark GSM8K (70.15% vs. 68.98%), confirming that the ability of most LLMs to evaluate the reasoning process of MWP is insufficient.
Point 3: The ability of most LLMs to evaluate the reasoning process of MWP is insufficient.

heading: 4.4 Incapable of Identifying Correct Solutions or Incorrect Solutions?
text: From the previous experimental results, we observe that most LLMs perform poorly on MCGSM8K and MCGSM8K-2Options. To figure out whether the model is incapable of identifying correct solutions or incorrect solutions, we propose a trueor-false-question benchmark GSM8K-judgement (Figure 2) aiming at directly judging the correctness of a solution. Specifically, for each question in GSM8K, we append a solution to the end of the question. We utilize the correct solution synthesized by the ground-truth answer and the answer rationale for constructing the positive sample. For one open-formed question, there are theoretically infinite numbers of incorrect solutions generated by the model with each one varying from others. To eliminate randomness, we design three various negative samples utilizing the three high-quality incorrect solutions from the options of MCGSM8K, thus resulting in a total of 1319 positive samples and 3957 negative samples. The model setting is the same as that in subsection 4.1.1. We utilize 5- shot in-context learning and CoT prompts to maximize accuracy, as shown in Appendix E Figure 5.
GSM8K-Judgement Model GSM8K MCGSM8K MCGSM8K -No-Rationale MCGSM8K -2Options TPR TNR
The testing accuracy of GSM8K-judgement is illustrated in the sixth and seventh columns of Table 1. As for general open-source models, the average True Negative Rate (TNR) is 76.68%, which is 34.71% higher than the average True Positive Rate (TPR) (41.97%) and 26.68% higher than the random-chance accuracy (50%). In general, it is easier for humans to identify incorrect solutions than correct solutions. This experimental result is consistent with human behavior. However, the performance of mathematical specialized models contradicts human performance, which are tuned on numerous augmented correct solutions. In contrast to general open-source models, the average TPR of mathematical specialized models is 40.01% higher, but the average TNR is 48.83% lower, which suggests that models merely fine-tuned on correct solutions can improve the ability to judge the correct reasoning process but greatly weaken the ability to judge the incorrect reasoning process. GPT-3.5Turbo also shows similar performance as mathematical specialized models. Only GPT-4 achieves remarkable TPR and TNR results at the same time. Point 4: Fine-tuning with only correct solutions improves the performance in evaluating correct solutions, but leads to a huge drop in evaluating incorrect solutions.

heading: 4.5 Fine-tuning
text: Finally, we explore whether fine-tuning can improve the ability of LLMs to evaluate the reasoning process of MWP. We use the most widely used model LLaMA-2-13B as the base model for finetuning. The LLaMA-2-13B is trained by fully finetuning on 8 NVIDIA A100 GPUs. The training details are following Yu et al. (2023).
In Table 2, the accuracy of SFT-GSM8K by finetuning the LLaMA-2-13B on the training data of GSM8K is extracted from RFT (Yuan et al., 2023). First, we fine-tune the base model on 6,000 training data from MCGSM8K and 3,000 training data from GSM8K-Judgement (1,500 positive samples
and 1,500 negative samples), respectively. Significant performance improvements are obtained for both SFT-MCGSM8K and SFT-Judgement on in-domain datasets (IND), demonstrating the effectiveness of the training data in improving mathematical reasoning process evaluation ability. Meanwhile, SFT-Judgement achieves an improvement of 32.75% on TPR and 8.33% on TNR, revealing that finetuning on training data from GSM8KJudgement is effective for enhancing capabilities in identifying both correct and incorrect solutions. On out-of-domain datasets (OOD) including GSM8K, MCGSM8K -No-Rationale, and GSM8KJudgement, the performance decline between SFTMCGSM8K and the base model can even reach up to 100%, which is consistent with the conclusion drawn from MAmmoTH (Yue et al., 2023) that fine-tuning LLMs using supervised data specific to certain datasets improves in-domain performance while reduces generalization to tasks beyond their fine-tuning data. Especially in the GSM8K benchmark, SFT-MCGSM8K and SFT-Judgement have completely lost their ability to follow instructions, resulting in irrelevant answers.
To maintain the generalization ability of the model in solving mathematical problems, we collect a hybrid fine-tuning dataset by mixing the training data from GSM8K, MCGSM8K, and GSM8KJudgement. On GSM8K, SFT-hybrid lags behind SFT-GSM8K. We speculate that the question form of MCGSM8K and GSM8K-Judgement is completely different from that of GSM8K, resulting in the ability to evaluate the mathematical reasoning process not being successfully transferred to solve the mathematical reasoning problems. Compared with the base model, the improvement of the SFThybrid in TPR, TNR, MCGSM8K, and GSM8K is 41.24%, 8.87%, 37.38%, and 16.62%, respectively. Moreover, for the unseen MCGSM8K-NoRationale task, the performance of the SFT-hybrid
is better than the base model (+7.43%). The results show that we can improve the ability of LLMs to evaluate the mathematical reasoning process of MWP by fine-tuning.

heading: 4.6 Case Study
text: Appendix F shows some examples generated by different models to solve MCGSM8K and GSM8KJudgement problems.

heading: 5 Conclusion
text: In this paper, we focus on exploring the ability of LLMs to evaluate the mathematical reasoning processes of MWP. To achieve this, we utilize incorrect solutions generated by multiple advanced LLMs to curate two benchmarks. One is MCGSM8K and its two variants, a new type of multiple-choice question dataset, in which each option contains a solution to solve the problem. The other one is GSM8K-Judgement, which judges whether a solution to a given problem is true or false. The poor performance of LLMs on MCGSM8K confirms the incapable ability of most LLMs in mathematical reasoning process evaluation. In particular, the performance on GSM8K-Judgement exhibits that it is easier to identify incorrect solutions. However, merely fine-tuning with correct solutions improves the performance in evaluating correct solutions, but leads to a huge drop in evaluating incorrect solutions. Fine-tuning models on the proposed training data greatly improves the mathematical process evaluation ability. Exploring the relation between the ability of mathematical problem-solving and mathematical reasoning process evaluation is left to future work.
Limitations
Through the above experiments and analyses, we summarize the following limitations:
1) In this work, we test the mathematical reasoning process evaluation ability of LLMs on limited benchmarks. In the future, we will utilize various MWP benchmarks, e.g., MATH and AQuA, to construct more comprehensive mathematical reasoning process evaluation benchmarks.
2) All incorrect solutions in the proposed benchmarks are generated by advanced LLMs, thus there are inevitably biases inherent in model generations. Furthermore, the constructed benchmark does not reflect the ability of LLMs to evaluate incorrect solutions written by humans.
3) The fine-tuned model does not exhibit a significant improvement in mathematical problemsolving. How to transfer the ability in mathematical reasoning process evaluation to mathematical problem-solving will be a future work.
Ethics Statement
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors. Informed consent was obtained from all individual participants included in the study.

heading: Acknowledgments
text: This work is supported by the National Key Research and Development Program of China under Grant 2023YFF1204901, the National Natural Science Foundation of China under Grant NSFC62076172, and the Key Research and Development Program of Sichuan Province under Grant 2023YFG0116.

heading: A The Impact of Different Prompts on Model Performance
text: First, we compare testing accuracy with/without Chain-of-Thought (CoT) prompting. We utilize
CoT prompting to make models solve math problems through step-by-step natural language descriptions, as shown in Figure 5. On the MCGSM8K benchmark, the average accuracy of all tested models with and without CoT prompting is 41.59% and 38.46%, respectively, exhibiting that CoT prompting brings slight improvements. By analyzing CoT answers illustrated in Figure 6 and Figure 7, we find that most models have difficulty in identifying incorrect computational processes and logical fallacies in reasoning steps, thus leading to incorrect results. In addition to fine-tuning, some welldesigned CoT prompts can also bring a certain degree of performance improvement.
On the MCGSM8K-No-Rationale benchmark, the accuracy with CoT prompting is significantly higher than the accuracy without CoT prompting for all tested models, e.g., the accuracy of LLaMA2-70B with and without CoT prompting is 58.45% and 37.60%, and the accuracy of MetaMath-70B with and without CoT is 76.04% and 37.30%.
On the GSM8K-Judgement benchmark, the performance gap with and without CoT prompting is negligible for general closed-source models. Meanwhile, CoT prompting brings significant performance improvements for LLaMA-2-70B. We can observe a huge performance decline between CoT prompting and no CoT prompting for mathematical specialized models, as they forget the instructionfollowing ability without CoT prompting, resulting in completely irrelevant answers.
Further, we compare the testing accuracy by different CoT prompts on MCGSM8K. We utilize two different prompts including describing the task and explaining the answer (P1), and explaining why other options are wrong and why the predicted option is correct before outputting the correct answer label (ours). We illustrate the results in Table 3. The performance improvement brought by differ-
ent prompts is less than 3%, and the best result so far is still much lower than that on GSM8K. We suspect this is mainly due to the model’s high confidence in the incorrect solutions.

heading: B Ablation Study on Data Construction
text: To keep the incorrect options in MCGSM8K confusing and diverse, we have designed three effective ways to select negative examples from candidate solutions, including random selection, similarity ranking, and PPL ranking. To explore the role of these three ways on the testing accuracy of the benchmark, we study the following setups:
(1) PPL Ranking (P): On the test data constructed by only PPL Ranking selection, LLaMA2-70B achieves an accuracy of 51.60%, which is 13.31% higher than that on MCGSM8K, as shown in Figure 3. We suggest that the single PPL ranking selection may cause the constructed negative examples with high textual similarity. Thus, the performance gain may not be the improved ability of the model to evaluate the reasoning process, but rather the model successfully selecting the correct solutions based on text similarity.
(2) Random Selection + PPL Ranking (R + P): The performance of LLaMA-2-70B has dropped slightly from 51.60% to 49.28%. This ablation reflects that multiple selection methods are more likely to produce diverse negative examples, which can lead to improved performance. However, the quality of the negative samples by random selection cannot be guaranteed, thus the performance improvement is relatively small.
(3) Random selection + PPL Ranking + Similarity Ranking (MCGSM8K): After mixing similarity ranking, the performance of LLaMA-2-70B has dropped the most, reaching a minimum accuracy of 38.29%. We suggest that incorrect solutions similar to the correct solution are most likely to confuse the model.

heading: C Principles for selecting LLMs to generate solutions
text: The reasons for choosing open-source LLMs and the inclusion of LLMs of relatively smaller sizes (i.e., 13B) to synthesize incorrect solutions are as follows:
Firstly, we need to emphasize that the stronger the model’s mathematical problem-solving ability, the lower the probability of using reject sampling to get the desired incorrect answers. For example, we
found that because some GSM8K questions were too easy for the strong models such as GPT4, and MetaMath-70B, we couldn’t get the desired wrong answer even to sample 50 times.
Secondly, we also tried to use prompt words to push chat models deliberately generating wrong answers, but we found some common patterns in the generated answers. Similarly, solutions generated from the same closed-source model also have similar characteristics and distributions, and these patterns are easily captured by LLMs, leading to a decline in the difficulty and quality of the benchmark.
Given the above factors, the efficiency of constructing incorrect solutions by closed-source models is low. To obtain a variety of diverse incorrect solutions for each question from GSM8K, we allow multiple LLMs with sizes of 13B and 70B including general LLMs Qwen, LLaMA-2, and WizardLM with stronger instruction following capabilities, and mathematical specialized LLMs MetaMath to generate incorrect solutions.

heading: D MCGSM8K Analysis
text: D.1 Statistics
Table 5 describes the basic statistics of MCGSM8K, which consists of a total of 1,319 multiple-choice questions with each option containing one solution.
D.2 Diversity
We conduct further analysis to examine the distinctions among incorrect options. For each sample, we calculate the ROUGE-L overlap between the three incorrect options and illustrate the distribution of these scores in Figure 4. The results reveal significant diversity among incorrect options.
D.3 Quality
So far, we have demonstrated the quantity and diversity of incorrect options, yet their quality remains uncertain. To address this concern, we randomly select 100 incorrect options from MCGSM8K. Subsequently, three expert annotators (the authors of this work) are tasked with assigning a score ranging from 1 to 5 to each incorrect option, in terms of fluency, relevance, validity, and consis-
tency. The evaluation results, presented in Table 6, indicate that the majority of incorrect options exhibit relevant and reasonable solutions to the given question, albeit some may contain a certain level of noise, such as repetition, calculation errors, etc. We use the intraclass correlation coefficient (CCI) to measure the correlation between raters. The correlation value calculated by ICC(C,1) is 0.92, higher than 0.9, which means that the evaluation of the three raters has a high degree of consistency.

heading: E Prompts
text: We illustrate some examples in Figure 5.

heading: F Case Study
text: F.1 MCGSM8K Case Study We illustrate the result in Figure 6.
F.2 GSM8K-Judgement Case Study We illustrate the result in Figure 7.

heading: Future Work
text: Exploring the relation between the ability of mathematical problem-solving and mathematical reasoning process evaluation is left to future work. Limitations
Through the above experiments and analyses, we summarize the following limitations:
1) In this work, we test the mathematical reasoning process evaluation ability of LLMs on limited benchmarks. In the future, we will utilize various MWP benchmarks, e. g. , MATH and AQuA, to construct more comprehensive mathematical reasoning process evaluation benchmarks. 2) All incorrect solutions in the proposed benchmarks are generated by advanced LLMs, thus there are inevitably biases inherent in model generations. Furthermore, the constructed benchmark does not reflect the ability of LLMs to evaluate incorrect solutions written by humans. 3) The fine-tuned model does not exhibit a significant improvement in mathematical problemsolving. How to transfer the ability in mathematical reasoning process evaluation to mathematical problem-solving will be a future work