MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs
Authors: Zhongshen Zeng
Yinhong Liu
Yingjia Wan
Jingyao Li
Pengguang Chen
Jianbo Dai
Yuxuan Yao
Rongwu Xu
Zehan Qi
Wanru Zhao
Linling Shen
Jianqiao Lu
Haochen Tan
Yukang Chen
Hao Zhang
Zhan Shi
Bailin Wang
Zhijiang Guo
Jiaya Jia
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models;Reasoning;System-2;Slow Thinking;Resource and Evaluation;Analysis and Interpretability
Primary area: evaluation
TL;DR: MR-Ben is a comprehensive process-oriented evaluation benchmark comprising 5,975 questions that cover various subjects, ranging from natural science to coding, challenging LLMs to score the reasoning steps of candidate solutions.

Abstract: Large language models (LLMs) have shown increasing capability in problem-solving and decision-making, largely based on the step-by-step chain-of-thought reasoning processes. However, evaluating these reasoning abilities has become increasingly challenging. Existing outcome-based benchmarks are beginning to saturate, becoming less effective in tracking meaningful progress. To address this, we present a process-based benchmark MR-Ben that demands a meta-reasoning skill, where LMs are asked to locate and analyse potential errors in automatically generated reasoning steps. Our meta-reasoning paradigm is especially suited for system-2 slow thinking, mirroring the human cognitive process of carefully examining assumptions, conditions, calculations, and logic to identify mistakes. MR-Ben comprises 5,975 questions curated by human experts across a wide range of subjects, including physics, chemistry, logic, coding, and more. Through our designed metrics for assessing meta-reasoning on this benchmark, we identify interesting limitations and weaknesses of current LLMs (open-source and closed-source models). For example, with models like the o1 series from OpenAI demonstrating strong performance by effectively scrutinizing the solution space, many other state-of-the-art models fall significantly behind on MR-Ben, exposing potential shortcomings in their training strategies and inference methodologies.

MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs
Zhongshen Zeng'
Yinhong Liu
Yingjia Wan
Jingyao Li'
Pengguang Chen'
Jianbo Dai
Yuxuan Yao
Rongwu Xu
Zehan Qi
Wanru Zhao?
Linling Shen
Jianqiao Lu
Haochen Tan
Yukang Chen
Hao Zhang
Zhan Shi
Bailin Wang"
Zhijiang Guo
Jiaya Jia"
'Chinese University of Hong Kong University of Cambridge Uniiveritt of Edinburgh CCity University of Hong Kong University of Hong Kong &Nanyang Technological University Massachuseets Institute of Technology
'Tsinghua University University of Texas at Austin
Abstract
Large language models (LLMs) have shown increasing capability in problem- solving and decision-making, largely based on the step-by-step chain-of-thought reasoning processes. However, evaluating these reasoning abilities has become increasingly challenging. Existing siccome-based benchmarks beginning saturate, becoming less effective in tracking meaningful progress. To address this, present process-based benchmark MR-Ben that demands meta-reasoning skill, where LMs are asked to locate and analyse potential errors in automati- cally generated reasoning steps. Our meta-reasoning paradigm is especially suited for system-2 slow thinking, mirroring the human cognitive process of carefully examining assumptions, conditions, calculations, and logic to identify mistakes. MR-Ben comprises 5,975 questions curated by human experts across a wide range of subjects, including physics, chemistry, logic, coding, and more. Through designed metrics for assessing meta-reasoning on this benchmark, we identify inter- esting limitations and weaknesses of current LLMs (open-source and closed-source models). For example, with models like the ol series from OpenAl demonstrat- ing strong performance by effectively scrutinizing the solution space, many other state-of- offie--f-theaatt art models fall significantly behind on IR-Ben, exposing potential shortcomings in their training strategies and inference methodologie
Introduction
Reasoning, the cognitive process of using evidence, arguments, and logic to reach conclusions, is erucial for problem-solving. decision-making, and critical thinking 165 19 With the rapid
advancement of Large Language Models (LLMMs), there is an increasing interest in exploring their
reasoning capabilities 330 [57]. Consequently, evaluating reasoning in LLMs reliably becomes paramount. Current evaluation methodologies primarily focus on the final result [16 28 22 60), disregarding the intricacies of the reasoning process. While effective some extent, such evaluation practices may conceal underlying issues like logical errors or unnecessary steps that compromise the
accuracy and efficiency of reasoning 66
"Correspondence Zhijiang Guo (rg283@cam.ac.uk) and Jiaya Jia (lcojia coe. h.t....c...kkccckkkk...k... hk) Our dataset and codes are available on httttp::/rraddolph-zeee gittgub..ttt....... io/Mr Ben. github........... io
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Therefore, it is important to complement outcome- based evaluation with an intrinsic evaluation of the quality of the reasoning process. However, current benchmarks for evaluating LLMs' reasoning capabilities have certain limitations in terms of their scope and size. For instance, PRM800K [38 categorizes each reasoning step as positive, negative, or neutral. Similarly, BIG-Bench Mistake focuses on identifying errors in step-level answers. We follow the same meta-reasoning paradigm as MRKGSMMK [77] and MR-Math [68], which go a step further by providing the error reason for the first negative step in the reasoning chain. However, these benchmarks are limited to narrower task Cope-MMR-SMMSS and MR-Math focus solely on mathematical reasoning, while BIG-Bench Mistake mainly assesses logical reasoning. To ensure comprehensive evaluation of reasoning abilities, it is crucial to identify reasoning errors and assess the LLMs' capacity to elucidate them across wider domains.
To bridge this gap, we construct a comprehensive benchmark MR-Ben comprising 6k questions covering a wide range of subjects, including natural sciences like math, biology, and physics, as well as coding and logic. One unique aspect of MR-Ben is its meta-reasoning paradigm, which involves challenging LLMs to reason about different forms of reasoning. In this paradigm, LLMs take on the role of a teacher, evaluating the reasoning process by assessing correctness, analyzing potential errors, and providing corrections, depicted in Figure
47 uncovers distinct limitations and previously unidentified weaknesses in their reasoning abilities. While many LLMs are capable of generating correct answers, they often struggle to identify errors within their reasoning processes and explain the underlying rationale. To excel under our meta-reasoning paradigm, models must meticulously scrutinize assumptions, conditions, calculations, and logical steps, even inferring step outcomes counterfactually. These requirements align with the characteristics of "System-2" slow thinking [35 91. which we believe remains underdeveloped in most of the state-of-the-art models we evaluated.
Our analysis of various LLMs 50 51
33
We suspect that a key reason for this gap lies in current fine-tuning paradigms, which prioritize correct solutions and limit effective exploration of the broader solution space. Echoing this hypothesis, we observed that models like ol -preview [52. which reportedly incorporate effective search and disam- biguation techniques across trajectories in the solution space, outperform other models by a large margin. Moreover, we found that leveraging high-quality and diverse synthetic data M significantly mitigates this issue, offering a promising path to enhance performance regardless of model size. Ad- ditionally, our results indicate that different LLMs excel in distinct reasoning paradigms, challenging the notion that domain-speente specific enhancements necessarily yield broad cognitive improvements. We hope that MR-Ben will guide rescarchers in comprehensively evaluating their models' capabilities and foster the development more robust Al reasoning frameworks.
Our key contributions are summarized as follows:
We introduced MR-Ben, which includes around 6k questions across wide range of subjects, from natural sciences to coding and logic, and employs a unique meta-reasoning paradigm.
We conduct an extensive analysis of various LLMs on MR-Ben, revealing various limitations and previously unidentified weaknesses in their reasoning abilities.
We offer potential pathways for enhancing the reasoning abilities of LLMs and challenge the assumption that domain-specific enhancements necessarily lead to broad improvements.
2 Related Works
Reasoning Benchmarks Evaluating the reasoning capabilities of L.MS is crucial for understanding their potential and limitations. While existing benchmarks often assess reasoning by measuring performance on tasks that require reasoning, such as accuracy, they often focus on specific reasoning types like arithmetic, knowledge, logic, or algorithmic reasoning. Arithmetic reasoning. involving mathematical concepts and operations, has been explored in benchmarks ranging from elementary
more complex and large-scale tasks 28 48). Knowledge reasoning. on the other hand, requires either internal (commonsense) or external knowledge, or a combination of
word problems [37 4 [6]
both [4 62 Logical reasoning benchmarks, encompassing deductive and inductive reasoning,
use synthetic rule bases for the former [5
61 [8] and specific observations for the latter to formulate general principles [78 7 Algorithmic reasoning often involves understanding the coding problem
description and performing multi-step reasoning to solve it [17 25 Benchmarks like BBH [59 and

---

Artthmetic Reasoning
Logical Reasoning
Algorithmic Reasoning
BM Instrucionn
Instruction def oxpecsrton, jumbettsaoorss
Instruction
intencee Wiendee J. Firter no intenenct
throght pintole faster than
was inardeeed be
"ecruummet has begunn and total of oa Maniiiiles have been selected Alce and Bob Calouatte the expested value
unterason
That yau mase Iealy:
of balowinge rright be rue?
Ac CO2
B: CHd
my wore eterwened. and
Questions
C. CSH12
D: CaH1s
Cnly and anoilerr cardicels naru inanniwed. keviowad
and five oiher candeaac wer
Shep Hescal Cinttant bew of atosion
Shep Calouiate meter menoss
Selution
day Solation
Stop 1:IFF was interiowedd then K must howe bever rerdeeeed strcw will not be undease snaeraaooed
WMae Solution
det sopedtation nunterrjezzeer Caltuuttie the ecpected sumber posiiin indand orderr
stap The moter moces cansiiaaee gaec
Bt's
CC22 mozar
gimz.
score in anuamenn anuumeeaaaccsoona
CoT Answers
sanokdate gas compared iil
MS moril lern Hntessowwal
loast condicanes
If Protaati By resume of soons Value appoara at sama postan T.M p. Sacoonaad
Cholis 00211
udon Cholce
Solvton: Choice
sad Analysia Error Stop: Slop 4
Analysis
sdd Analysis
Error Reason: The reñoo should be the eq par rost of unnknown ges ooer tellum besed on the Cratemn' low of aftirin.
Step: Stop
Error Reason: The probabiltty of resume of scoore sppears postion should be 1ik instedd of 1
Error Step: Line 16
possible combinaat iiinations candilates
Error Analysis
Correction: "Choico A. 1/44/0) 3.52" 3.5.2"
which
Mi."
Correction:" "p 1/K"
Figure I: Overview of the evaluation paradigm and representative examples in MR-Ben. Each data point encompasses three key elements: a question, a Ofain-of-Thhugeee (CoT) answer, and an error analysis. The CoT answer is generated by various LLMs. Human experts annotate the error analyses, which include error steps, reasons behind the error, and subsequent corrections. The three examples shown selected to represent arithmetic, logical, and algorithmic reasoning types.
indirectly assess reasoning by evaluating performance on tasks that require it. However, these benchmarks primarily focus on final results, neglecting the analysis of potential errors in the reasoning process. Unlike prior efforts, MR-Ben goes beyond accuracy by assessing the ability to locate potential errors in the reasoning process and provide explanations and corrections, Moreover, MR-Ben covers different types of reasoning, offering more comprehensive assessment.
MMLU 27
Evaluation Beyond Accuracy Many recent studies have shifted their focus from using only the
final result to evaluating the reasoning quality beyond accuracy. This shift has led to the development of two approaches: reference-fres and reference-based evaluation. Reference- methods aim to assess reasoning quality without relying on guman--roovded solutions. For example, ROSCOE [23]
evaluates reasoning chains by quantifying reasoning errors such as redundancy and hallucination. Other approaches convert reasoning steps into structured forms, like subject-verb-ojeect frames (56) or symbolic proofs [58], allowing for automated analysis. Reference- based methods depend on human- generated step-by-ster solutions. For instance, PRM800K (38 offers solutions to MATH problems 288. categorizing each reasoning step as positive, negative, or neutral. Building on this,
MR-GSM8K 77 and MR-Math [68] further provide the error reason behind the first negative step. MR-GSM8K focuses on elementary math problems, sampling questions from GSM8K (16]. MR-Math samples a smaller set of 459 questions from MATH [28] Using the same annotation scheme, BIG-Bench Mistake [64] focuses on symbolic reasoning. encompasses 2, 28186 instances from 5 tasks in BBH [59]. Despite the progress made by these datasets, limitations in scope and size remain. To address this, we introduce MR-Ben, a benchmark consisting 5,975 manually annotated instances covering wide range of subjects, including natural sciences, coding, and logic. MR-Ben also features more challenging questions, spanning high school, graduate, and professional levels.
3 MR-Ben: Dataset Construction
3.1 Dataset Structure
To comprehensively evaluate the reasoning capabilities of L.LMs, MR-Ben employs meta-reasoning paradigm. This paradigm casts LLMs in the role of teacher, where they assess the reasoning process
3

---

by evaluating its correctness, analyzing errors, and providing corrections. As shown in Figure | each data point within MR-Ben consists of three key elements: question, CoT answer, and an error analysis. The construction pipeline is shown in Figure 6 in Appendix
Question The questions in MR-Ben are designed to cover diverse range of reasoning types and difficulty levels, spanning from high school to professional levels. To ensure this breadth, we curated questions from various subjects, including natural sciences (mathematics, biology, physics), coding, and logic. Specifically, we sampled questions from mathematics, physics, biology, chemistry. and medicine from MMLU [27]. which comprehensively assesses LLMs across academic and professional domains. For logic questions, we draw from LogiQA [40, which encompasses broad spectrum of logical reasoning types, including categorical, conditional, disjunctive, and conjunctive reasoning. Finally, we select coding problems from MHPP [17. which focuses on function-level code generation requiring advanced algorithmic reasoning. Questions in MMLU and LogiQA require a single-choice answer, while MHPP requires snippet of code as the answer.
CoT Answer We queried GPT-3.5- 1...5--urb-0222 [50]. Claude2 [5] and Mestral-eedium 322 (as
of February 2024) using prompt template (provided in Figure Zi Appendix designed to elicit step-by-step solutions [66]. For clarity, all LLMs were instructed format their solutions with numbered steps, except for coding problems. To encourage diverse solutions, set the temperature parameter to 1 during sampling. This empirical setting yielded satisfactory instruction following and desirable fine-gen ine-grained reasoning errors, which annotators and evaluated models are expected to identify.
3.2 Annotation Process
After acquiring the questions and their corresponding Chain-of-Thoughe (CoT) answers, we engage annotators to provide error analyses. The annotation process is divided into three stages.
Answer Correctness CoT answers that result a final answer different from the ground truth are automatically flagged as incorrect. However, for cases where the final answer matches the ground truth, manual annotation is required. This is because there are instances where the reasoning process leading to the correct answer is flawed, illustrated in the middle example of Figure Therefore, annotators are tasked with meticulously examining the entire reasoning path to determine the correct final answer is direct result of the reasoning process.
Error Step This stage is applicable for solutions with either an unmatched final output or a matched final output underpinned by flawed reasoning. Following the prior effort (38] each step in the reasoning process is categorized as positive, neutral, or negative. Positive and neutral steps represent stages where the correct final output remains attainable. Conversely, negative steps indicate a divergence from the path leading to the correct solution. Annotators are required to identify the first step in the reasoning process where the conditions, assumptions, or calculations incorrect, making the correct final result unreachable for the subsequent reasoning steps.
Error Reason and Correction Annotators are tasked with conducting an in-depth analysis of the reasoning that led to the identified error. As shown in Figure annotators are required to provide the error reason and the corresponding correction to this reasoning step. This comprehensive approach ensures a thorough understanding and rectification of errors in the reasoning process.
3.3 Data Statistics
Table presents the statistics of MR-Ben. The benchmark exhibits a balanced distribution of correct and incorrect solutions, with an overall correct solution rate of 40.3%. Solutions, on average, involve 9.5 steps, and errors typically manifest around the fourth step (4.5). The questions and solutions are substantial, with average lengths of 85.6 and 308.8 words, respectively. The subject-wiss analysis reveals that Math is the most challenging, with a correct solution rate of mere 16.2%. This could be attributable to the intricacy the arithmetic operations involved. Conversely, Biology emerges as the least daunting, with high correct solution rate of 59.6%. Coding problems have the longest solutions, averaging 950.3 number of words. This underscores the complexity and the detailed procedural reasoning inherent in coding tasks. Similarly, Logic problems have the longest questions, averaging

---

Table 1: Statistics of MR-Ben. The length of questions and solutions are measured inthe number of words. Notice that the steps for coding denote the number of lines of code. They are not directly comparable with other subjects.
Math
Medicine
Biology
Physics
Chemistry
Logic
Coding
Total
Pnestion-Solution Pairs
918
828
1035
667
848
1441
238
5975
Correct Solution Ratio
16.2%
31.0%
59.6%
47.8%
45.0%
51.1%
31.1%
40.3%
Avg Solution Steps
6.8
5.3
5.1
5.7
5,6
5.3
32.5*
9.5
Avg. First Error Step
3.1
3.0
2.7
3.1
3.0
2.8
14.0
4.5
Avg Length of Questions
44.3
88.7
56.3
66.6
48.1
154.8
140.1
85.6
Avg Length of Solutions
205.9
206.1
187.6
199.4
194.5
217.7
950.3
308.8
154.8 words. This is in line with the need for elaborate descriptions in logical reasoning. The typical step at which the first enror occurs is fairly consistent across most subjects, usually around the 3rd step out of a total of However, Coding deviates from this trend. The first error tends to appear earlier, specifically around the 14th line out of a total of 32.5 lines. This suggests that the problem-solving process in Coding may have distinct dynamics compared to other subjects.
3.4 Quality Control
Given the complexity of the questions, which span range of subjects from high school to professional levels, we enlisted the services an annotation company. This company meticulously recruited annotators, each holding minimum of bachelor's degree. Before their trial labeling, annotators are thoroughly trained and are required to review the anmotation guidelines. We've ve included the guidelines for all subjects in Appendix H for reference. The selection of annotators is based on their performance on a balanced, small hold-out set of problems for each subject. In addition to the annotators, team of 14 quality controllers diligently monitors the quality of the annotation weekly. As final layer of assurance, we have meta controllers who scrutinize the quality of the work.
Annotators
Quality Assurance Every problem in MR-Ben undergoes a rigorous three-round quality assurance process to ensure its accuracy and clarity. Initially, each question is labeled by two different annotators. Any inconsistencies in the solution correctness or the first error step are identified and reviewed by quality controller for arbitration. Following this, every annotated problem is subjected to secondary review by annotators who were not involved in the initial labeling. This is to ensure that the annotations for different solutions to the same problem are consistent and coherent. In the final phase of the review, 10% of the problems are randomly sampled and reviewed by the meta controllers. Throughout the entire evaluation process, all annotated fields are meticulously examined in multiple rounds for their accuracy and clarity. Any incorrect annotations or those with disagreements are progressively filtered out and rectified, ensuring a high-quality dataset. This rigorous process allows us to maintain a high level of annotation quality.
Dataset Artifacts & Biases Table relatively balanced distribution of correct and incorrect solutions. However, an exception was observed in mathematical subjects, where the distribution tends to skew towards incorrect solutions. This skew could suggest inherent complexity ambiguity in mathematical problem statements. Our analysis of the first error step across all subjects indicated that errors predominantly occur in the initial stages (n S 7) of moblemm-solving and are distributed relatively uniformly. This pattern was consistent across most subjects, with no significant skew towards later steps. More detailed discussions of biases provided in the Appendix C
4 Evaluation
For each question- question-solution pair annotated, the evaluated model are supposed to decide the correctness of the solution and report the firs-eeroo--seep and error-reason if any. The and first-errorsseep is scored automatically based on the manual annotation result. Only when the evaluated model correctly identified the incorrect solution and first-error-seep will its error-reason be further examined manually automatically by models. Therefore in order to provide unified and normalized score to reflect the overall competence of the evaluated model, we follow the work of 7 and apply metric named MR-Score, which consist of three sub-metries.
correctness
5

---

The first one is the Matthews Correlation Coefficient (a.k.a MCC, 46) for the binary classification of
TP TN FP FN
MCC= = V(TP+FP) x (TP PN) x
(I)
(TN + FN)
where TP, TN, FP, EN stand for true positive, true negative, false positive and false negative. The MCC score ranges from -1 to +1 with 1 means total disagreement between prediction and observation, O indicates near random performance and +1 represents perfect prediction. In the context of this paper, we interpret negative values as no better than random guess and set O cut-off threshold for normalization purpose.
The second metric is the ratio between numbers of solutions with correct first-crror-stey predicted and the total number of incorrect solutions.
N fis crrorsrsrerrreeepppppppppp
ACCstep
Nincorret_soll sols
The third metrics is likewise the ratio between number of solutions with correct first-error-step plus correct error-reason predicted and the total number of incorrect solutions.
N correct_ eeror_rrreeeeee reason
ACCreason reason
(3)
N Wincorroc_sso solls
MR-Score is then a weighted combination of three metrics, given by
R-Sccore = w1 max(0, MCC) + w2 ACC'ssp + w3 * ACCressn
(4)
For the weights W1, W2 and W3, they are chosen based on our evaluation results to maximize the differentiation between different models. It important to note that the Matthews Correlation Coefficient (MCC) and the accuracy of locating the first error step can be directly calculated by comparing the responses the evaluated model with the ground truth annotations. However, assessing the accuracy of the error reason explained by the evaluated model presents more complexity. While consulting domain experts for annotations a feasible approach, we instead utilized GPT-4-Turbo as proxy to examine the eиror reasons, detailed in Figure II in Appendix D
We operate under the assumption that while our benchmark presents significant challenge for GPT-4 in evaluating complete solution correctness ieet--ddnnnnnnnnnneeeee the first error step and explaining the error reason it comparatively easier for GPT-4 assess whether the provided error reasons align with the ground truth. Specifically, in a hold-out set of sampled error reasons, there was a 92% agreement rate between the manual annotations by the authors and those generated by GPT-4. For more detailed evaluations on the robustness of MR-Score and its design thinking, please refer to our discussion in Appendix B
Experiments
5.1 Experiment Setup
To evaluate the performance of different models on our new benchmark, we selected diverse array of models based on size and source accessibility [ This included smaller models like Gemma-28 63]
Qwen @wenl.5-1.8B 1.8BB 77,, as well as larger counterparts such as Llama3-70B [47], Deepseek- 67B 10], and Qwen 1-5-7BBBBBB 77] We also compared open-source models (e.g. models from the
P,
Llama3 and Qwen 5/QWen2 series) against closed-source models from the GPT [5T], Claude (6)
Mistral 32). GLM (3) Yi [39], Moonshot (2), Doubao [12) families. Additionally, models from the
Deepseek-Codee [10] series were included to assess the impact of coding-focused pretraining on reasoning performance.
Given the complexity of our benchmark, even larger open-source models like Llama3-700 Instructt struggle to produce accurate evaluation results without the use of prompting methods, often achieving
Note: All models used in our experiments are instruction-fine finetuned versions, although this is not indicated in their abbreviated names

---

Table 2: Evaluation results MR-Ben: This table presents detailed breakdown of each model's performance evaluated under metric MR-Score across different subjects, where K stands for the number of demo examples here.
Bio.
Phy.
Math
Chem.
Med.
Logic k=1
Coding k=
Avg.
Model
k=
0=
1=
k=
1=
k=
k=1
k=1
Sloucd-Source LLMs
Claude3-Haiku
5.7
5.8
3.5
3.1
6.4
2.0
2.0
1.2
9.0
4.4
GPT-3.5-Turb
6.6
6.7
544
6.7
3.6
4.4
1.7
4.5
3.0
4.0
Doubao-pro-4k
8.4 13.5
10.0
11.7
12.3
15.5
10.6
17.5
5.9
10.0
4.5
5.5
9.8
7.4
8.8 11.6
Mistral-Large
22.2 28.0
26.7
25.4
24.3
28.2
24.0
27.0
15.9
19.3
14.7
17.1
21.1
21.4
21.3
23.8
Yi-Large
35.3. 40.7
37.2
36.8
36.5 20.6
40.0 39.1
29.3
32.1
25.1
31.3
21.9
25.7
32.2
32.3
Moonshot-vl--8
35.0 36.8
33.8 33.8
34.9 33.0
36.7 35.0
29.4 32.3
25.0
29.2
32.7
31.2
32.5 33.0
GPT-4o-mini
37.7 38.9
38.5 37.4
44.4 40.4
39.2 37.0
33.9 25.1
23.6 17.7
41.6 34,9
37.0 33.1
M--
46.2
42.5
38.4
43.1
34.5
37.5 32.5
39.4
GPT-4-Turb
44.7 47.3
42.8 45.2
44.3 45.4
44.0 46.0
38.8 38.
34.1 33.6
57.3
43.2 44.7
GPT-40
48.3 49.1
45.5 48.2
42.6 41.3
48.2 49.1
47.9 47.7
31.9 28.4
56.5 54.6
45.8 45.5
ol-mini
45.8 46.9
56.0
53.8
68.5 67.0
55.2 56.1
45.9 47.2
30.7 28.7
55.1 53.6
51.0 50.8
ol-prrview
54.1 56.0
62.2 61.7
69.8 70.3
60.6 60.3
54.3 55.1
46.1 45.3
65.1 70.0
58.9 59.8
Open-Source Small
Qwenl.5-1..
0.0
0.0
0.0
0.0
0.0
0.1
0.0
0.1
0.0
0.0
0.0
0.1
0.0
0.0
0.0
0.0
Gemma-2B
0.1
0.0
0.0
0.0
0.0
1.0
0.1
0.0
0.0
0.4
0.0
0.2
0.7
0.0
0.1
0.2
Qwen2-1.5B
2.2
2.8
2.2
1.3
6.3
3.3
2.9
11.2
1,5
0.0
2.1
Phi3-3.8B
12.7
13.3
16.4
8.4
9.1
10.2
Open-Source LLMs Medium
GL---49B
4.4
2.4
1.2
4.7
2.9
2.3
2.5
6.7
DeepSeck-7B
5.7
6.2
4.7
2.6
5.2
4.9
1.6
3.0
3.8
0.0
3.7
Deepseek- Coddee
7.4
5.5
7.8
5.6
7.2
8.6
7.4
6.0
5.5
4.6
6.7
8.4
4.9
7.0
6.3
DeepSeck-Codree
10.5
9.9
11.8
9.6
11.8
12.1
12.3
11.9
10.4
11.0
9.8
10.7
5.0
5.8
10.2
10.2
LLaMA3-8B
12.0
11.9
10,9
7.5
15.0
9.0
12.6
12.7
9.3
8.0
9.4
9.6
15.8
10.0
12.2
9.8
Yi-1.5-9B
10.4
14.8
11.9
12.9
12.5
15.6
13.1
14.4
9.5
14.8
9.1
9.5
4.8
6.3
10.2
12.6
Open-Source LLMs Large
Qwenl..-72B 15-BBB
15.3 19.2
12.9
13.6
12.0
10.0
13.9 16.3
11.7
14.7
10.4 12.9
3.9 5.9
11.5 13.3
DeepSeek-67B
17.1 19.7
14.9
17.3
15.4
16.2
16.3 20.6
14.7
12.2
13.6 14.3
14.5
15.2
15.2 16.5
LLaMA3-70B
20.4 27.1
17.4
20.5
14.9
19.5 25.1
16.3 19.3
16.3 16.8
29.8
16.7
19.2 20.2
DeepSeek-Ve V2-236B
30.0 37.1
32.2
36.5
32.2
32,5
26.5 32.4
27.4
27.1
32.3
Qwen2-72B
36.0 40.8
36.7 40.9
38.0 38.7
37.2
38.8
28.3
29.3
25.6
20.5
31.3
30.4
33.3 34.2
MR-Scores near zero. Consequently, employed step-wise chain-of-thouge prompting technique similar to those described in [77 64 This approach guides models systematically reasoning through solution traces before making final decisions, detailed in Appendix
Considering the complexity of the task, which includes question comprehension, reasoning through the provided solutions, and adhering to format constraints, few-shot demonstration setups are also explored to investigate models can benefit from In-Context Learning (ICL) examples. Due the context token limits, we report zero and one-shot results in the main result table (Table 2 The performance of additional few-shot configurations on a selection of models with various capabilities further discussed in Section 6.1
5.2 Experiment Results
The MR-Ben benchmark presents significant shift in the challenge for state-of-the-ar large language models, transitioning from question-answering the nuanced role of question-solutie scoring. This section details our findings, emphasizing variations in model performances and their implications.
Overall Performance Among the evaluated models, ol -preview consistently achieves the highest MR-Scores across all subjects, significantly outperforming most competitors from both open and
For the breakdown performances of models in the sub-tasks, please refer Table

---

MR MR--Scores of Models on refferre Reasoring Paradigms
DeepSeekv2 Qwen2-728 GLM-4
-te GPTT44-turba
Ol-Preview
hanagers
knowledge
math
logic
artthmetic
chemistry
algorithiiic
physics
0.4
biology
0.2
medicine
logic
coding
0.0
DeepSeck
Previow
Qwen2
Models
Figure 3: Model performance on different reason- ing paradigms
Figure 2: Model performance across subjects
closed-source communities. Notably, the open-sourced Qwen2-72B and Dpepseek-V2-22955 models are performing exceptionally well, surpassing every other open-sourced model including Llama3 by large margin. Their scores are even comparable to or greater than some of the most capable models from commercial companies, such as Mistral, Yi, and Moonshot Al. In the small language model category, the performance of Phi3-3.8B exceeds many of the mid-size models, including Baaeeek--Coee---e- whose size is around tenfold larger.
Performance across Model Size and Reasoning Paradigm Table reveals a general trend where larger models tend to perform better, highlighting the correlation between model size and the efficacy in complex reasoning tasks. However, this relationship is not strictly linear, as demonstrated by models like Phi3-3.8B, which excel despite their smaller size. Since MR-Ben challenges the language models to reason about the reasoning in the solution space among diverse range of domains, models like Phi-3 that are trained with effective data synthesis techniques and broader coverage of the solution space, intuitively achieve higher MR-Score. This suggests that while larger model sizes generally yield superior performances, techniques like knowledge distillation can also significantly boost reasoning performance. Similarly, although the size of the ol model series remains undisclosed, these models reportedly employ mechanisms that scale computation efficiently through effective exploration, frequent retrospection, and meticulous reflection within the solution space. These characteristics align closely with the principles of "system-2" thinking, which emphasizes deliberate, reflective problem-solving. As result, the ol models demonstrate more effective reasoning process, achieving significantly higher MR-Scores than other models by large margin.
Performance across Reasoning Types Our categorization into four reasoning types-knowleedge arithmetic, algorithmic, and logic-illustrates the unique challenges each model f'aces within these paradigms (Figure3) 3). Logic reasoning emerges as the most formidable due to the intricate logical operations required by questions from the LogiQA dataset. In stark contrast, ol o1-Preview and GPT-4- turbo demonstrate exceptional prowess in algorithmic reasoning, where their capabilities markedly surpass other models. Notably, models excel in different reasoning paradigms, reflecting their varied strengths and training backgrounds. For instance, despite Deepseek-Code Coder's s specialized pre-training focused on coding tasks, it does not necessarily confer superior abilities in algorithmic reasoning, underscoring that targeted pretraining does not guarantee enhanced performance across all reasoning types. Comparing the performance of the Deepscek-Codee with that of the Phi-3 model, which excels despite its much smaller size, highlights the potential significance of high-quality synthetic data in achieving broad-based reasoning capabilities.
Sensitivity to Task Difficulty and Solution Length An examination across educational levels shows most models perform better at high school-level questions than college-level ones, indicating an intuitive level of sensitivity to the difficulty levels of the questions. Additionally, our analysis finds a minor negative correlation between the length of solution steps and MR-Scores, as detailed in Figure 4 and Figure
8

---

Mil-Scores of Models on Different Bifficulty Levels
Impact of Solution Length on Mil-Scores by Models
0.7
0.6
Dificuttee
high school
college
0.5
0.5
Score
0.4
MR
0.3
0.2
0.1
DeepSeeke12
DI-Freciedl
GIN-4
GPTsduube
Qowen-722 285
0.0
0.0
DerpSeekvv2
GPT-4-turbo
Prevlew
Owen2-72 728
GLN-4
<=5
1-13
Models
Solution Step Numbers
Figure 4: MR-Scores of different models on dif- ferent levels of difficulty
Figure 5: The MR-Scores of models on solutions with different step numbers.
Summary: MR-Ben effectively differentiates model capabilities, often obscured in simpler settings. It not only identifies top performers but also underscores the influence of model size outcomes, while demonstrating that techniques like knowledge distillation and test-time compute scaling, as seen with the Phi-3 and ol models, can notably enhance smaller models' performance, challenging the dominance of larger models. The analysis further reveals that specialized training, such as in coding, does not guarantee superior algorithmic reasoning. This suggests the potential need for more balanced data approaches improved data synthesis methods.
Further Analysis & Discussion
6.1 Few Shot Prompting
As previously discussed and exemplified by our prompt template (Figure TO Appendix our evaluation method is characterized by its high level of difficulty and complexity. In this experiment, we aimed to determine whether providing a few step-wise chain-of-thought (CoT) examples could improve model performance in terms of format adherence and reasoning quality. The results, as presented in Table in Appendix, do not show a consistent pattern as the number of shots increases. While smaller language models like Gemma-2B exhibit performance improvements with additional shots, the performance of larger language models tends to fluctuate with an increasing number of shots. We hypothesize that for complex tasks, the lengthy few-shot demonstrations may act more as a hindrance, providing distracting information rather than aiding in format adherence and reasoning. Our empirical findings suggest that one-shot demonstration strikes the optimal balance between providing guidance and minimizing distraction. This supports our decision to focus on zero-shot versus one-shot comparisons our primary experiments, as detailed Table
6.2 Self Refine Prompting
As suggested by B1. large language models typically cannot perform self-correction without external ground truth feedback. To explore whether this phenomenon occurs in benchmark, adopted a similar setting by prompting the language model to verify its own answer across a three-round interaction sequence: query, examine, and refine. Our prompting template, detailed in Figure 8 Appendix D is minimalistic and designed solely to encourage the model to self-examine.
The results of this self-refinement process recorded in Table Notably, models smaller than Llama3-70B exhibit performance degradation with self-refinement, while larger models, such as GPT-4, show marginal benefits from the process. Conversely, from Llama3-8B to Llama3-70B, despite a significant portion correct predictions shifting to incorrect ones, as previously reported by [II, our benchmark shows increasing trend of incorrect predictions shifting correct ones as model size increases. This shift results in the significant performance improvements observed in models like Llama3-70B.
To understand the disproportionate improvement observed in the 70B model, we analyzed per- formance breakdown at the task level. These results are visualized and discussed in Figure 9 of

---

Table Comparison of average accuracy in identifying the first eπror step and the corre- sponding error reason, with and without prior knowledge of the solutions' correctness.
Table 4: Comparison of prompting methods: MR-Scores achieved by zero-shot step-wise CoT and Self-Refine technique.
Defection Acc. Reason Acc.
Model
O-shot CoT
Self-Refine
Model
w/o
with
w/o
with
Gemma-2B
0.1
0.2
Gemma-2B
0.3
0.1
0.1
0.0
Llama3-8B
11.7
11.3
Llama3-8B
15.5
26.4
6.6
11.9
Llama3-70B
17.7
27.5
Llama3-70B
14.5
34.6
9.1
25.7
GPT-4-Turbo
43.2
45.5
GPT-4-Turbo
40.9
41.6
37.9
38.0
Appendix E In short, believe the lack of consistency does not necessarily indicate more robust or advanced reasoning ability, despite the increase of the evaluation results.
6.3 Solution Correctness Prior
To verify the influence of external ground truth signals, we sampled 100 incorrect solutions from each subject respectively as our test set. By observing the same set of language models under a zero-shot CoT setting, aim to determine whether the knowledge of the solution's 's incorrectness enhances their ability to identify the first error step and the reason for the error.
The results in Table I that the benefits of knowing the solution correctness prior generally increase with the model's 's competence but begin to plateau at the level of sophisticated models like GPT-4. Specifically, the Gemma-2b model struggles significantly in our benchmark, showing nearly zero performance due to its limited ability to follow formats and comprehend complex tasks. Consequently, having the solution correctness prior does not improve its performance metrics. In contrast, models with moderate capabilities benefit substantially from this prior knowledge, which aids in accurately locating the first error step and elucidating the error reason. However, as model capabilities improve, the incremental benefits of this prior knowledge quickly diminish. For instance, GPT-4 shows only marginal improvement in identifying the first error step and almost negligible impact on error reason analysis when provided with the prior.
7 Conclusion
This paper highlights the importance of evaluating the reasoning capabilities of LLMs with process- oriented design and presents a comprehensive benchmark called MR-Ber that addresses the limitations of existing evaluation methodologies. MR-Ben consists of questions from diverse range of subjects and incorporates meta-reasoning paradigm, where LLMs act as teachers to evaluate the reasoning process. Our evaluation of diverse suite of LLMs on MR-Ba AR-Ben reveals several key limitations and weaknesses. Many models struggle with identifying and correcting errors within reasoning chains, demonstrating difficulty in performing system-2 style thinking such as scrutinizing assumptions, calculations, and intermediate steps. Furthermore, even state-of-the-art models often fail to maintain consistency across reasoning paradigms, exposing gaps in their generalization abilities. Additionally, our findings emphasize the importance of searching and reflecting on the solution space during inference. Models like the ol series showcase the potential of scaling test-time computation, where frequent retrospection and iterative search through multiple solution paths significantly enhance reasoning performance. Nevertheless, improving LLMs' reasoning abilities on complex and nuanced tasks remains open research question, and we encourage future work to develop upon MR- RR-Ben.
8 Acknowledgement
This work was supported in part by the Research Grants Council under the Areas of Excellence scheme grant AoE/E-601/22-R.
10

---

References
[1] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
[2] Moonshot AI. Moonshot ai, 2024. URL
cn/
[3] Zhipu AL. Welcome to glm-4, 2024. URL
cn/
[4] Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedzionsse Yejin Choi, and Hannanch Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation based formalisms. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties:
Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pages 2357-2367. Association for Computational Linguistics, 2019, doi: 10.18653/V1/N1999 1245. URL /ttp::////oo.og//1...o 18653/11/ n19- 1245
[5] Anthropic. Claude 2, 2024. URL htttttss:/wwww http:p://www..aatrrppp...oo/////
[6] Anthropic. Introducing the next generation of claude, 2024. URL httpp::/www.nndrropi. anthropic
com wwwpw//wwwwweeeeeeeeeeee
[7] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. (tti:2309//609, 2023.
[8] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli. Dustin Li. Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse. Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage. Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer EI Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield- -oodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: harmlessness from Al feedback. CoRR, abs/221 1 2.08033, 2022. doi: 10. 48550/arXiv arxiv. www...221.............. 22122.80033
48550/ARXXV.2212..0855 22000333...
URL
[9] Yoshua Bengio. Deep learning for system processing. Presentation at the AAAI-20 Turing Award Winners 2018 Special Event, February 9 2020.
[10] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao. Ying He, Wenjie Hu, Panpan Huang, Erhang Li. Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, Alex X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie. Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang, Bingxuan Wang, Peiyi Wang, Shiyu Wang, Yaohui Wang, Yongji Wang. Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yiliang Xiong, Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang. Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghua Zhang, Wentao Zhang. Yichao Zhang, Chenggang Zhao, Yao Zhao. Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. Deepseek LLM: scaling open-source language models with longtermism. CoRR. abs/2401 022944. 2024. doi: (0448500AAXX..240000095 URL https //doi.org/10. /005//////0//////////////// arXiv 029544
[11] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M.
11

---

Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005. 14165, 2020. URL. http::://aaxvvvvvo bbrrg/abs//0 abt/ 2005 14165
[12] Bytedance. Doubao team crafting the industry's most advanced Ilms., 2024. URL https www.doubao.. com/ cchat/
[13] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling
language models. CoRR, abss2210.111416, 2022. doi: URL https:://o/// org/ wtt.///....//////////////////.
10.4 1..4850/ARXI..210011444
48550/ http:/rxxiv....2 11416
[14) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the A12 reasoning challenge. CoRR, abs/ 2018. (tth:://///ati org aa0///0////10/. 000/1/03330007777
[15] Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. In Christian Bessiere, editor, Proceedings of the Twenty-Niuth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3882-3890. ijcai.org, 2020. doi: 10.24963/ICAA22220/533 URL https://doi. org/ 10. 24963/ http///0aaa.222/553333
[16] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jeny Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/21 10. 14168, 2021. URL https://arx/::
org/ abs /210. 14168
[17] Jianbo Dai, Jianqiao Lu, Yunlong Feng, Rongju Ruan, Ming Cheng. Haochen Tan, and Zhijiang Guo. Mhpp: Exploring the capabiliiiiss and limitations of language models beyond basic code generation. arXiv preprint 24XX::24005.143000 2024.
[18] Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pi- patanangkura, and Peter Clark. Explaining answers with entailment trees. In Marie-Francing Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 7358-73777 Associa- tion for Computational Linguistics, 2021. doi: 10. 18.18853/V W00.22.EENP--MAI..... URL https:://da org 100 10...011651/1112222
emnlp- mapp-mmain..m 585
[19] Ronald Fagin and Joseph Y. Halpern. Reasoning about knowledge and probability. J. ACM, 41(2):340-367, 1994. doi: 10. 10.1144//7466221746688 URL https://doi. /0t./.1.1154
774652.144655
[20] Patrick Fernandes, Aman Madaan, Emmy Liu, António Farinhas, Pedro Henrique Martins, Amanda Bertsch, José G. C. de Souza, Shuyan Zhou, Tongshuang Wu, Grahan Neubig, and André Martins. Bridging the gap: A survey on integrating (human) feedback for natural language generation. CoRR, abs/2305.00955 2023. doi: 10. 1044850/ARXIV2235500055 URL Inttps://do.. org/10. (erg/10.455000 48550/ arXiv. 2305 00955
[21]
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao. Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. RARR: researching and revising what language models say. using language models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), ACL 2023, Toronto, Canadu, July 9-14, 2023, pages 16477-16508, Association for Computational I. Linguistics, 2023. doi:
URL. httpp:::ts... http:://toooo.....o.e 18653/11/
a013.ac1- long. 910
12

---

[22] Mor Geva, Daniel Khashabi, Elad Segal. Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? question answering benchmark with implicit reasoning strategies. Trans. Assoc. Comput. Lingwistics, 9:346-361, 2021. doi: 10.1 162/TACIL
URL
httpss//doi. http////..rrg//..1////////////// aa01____00000 00370
[23] Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi and Asli Celikyilmaz. ROSCOE: A suite of metrics for scoring step-by-step reasoning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL wttph:ttp/////tc http:s:://opprreeiieeeeee net/
pdF7id=x\1JRpzZtss)
[24] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. CRITIC: large language models can self-correct with tool-interactive eritiquing. CoRR, abs/2305. 1738, 2023. doi: 10:4850/ARXIV2305.11 11738.
URL https://doi. org/10
(8650/arXiv.. 2305 11738
[25] Alex Gu, Baptiste Rozière, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. In Forty- first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 2/-27, 2024. OpenReview.net, 2024. URL htt.httts:///pppeeeeeeee
net/ wttp://.eeeffrrrre weee//oouum.ddFfpg522ssss
[26] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang. Séhastien Bubeck, Ronen Eldan, Adam Tau- man Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. CORR, abs/2306. 1144, 2023. doi: (0.48550/ARXI.22555 1644. URL.ttps:///oli org/ro..//.o 10. 48550/ arXiv 2306 11644
[27] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:://pppnreeiieeee
[28] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sii-Kit Yeung, editors, Pro- ceedings of the Neural Information Processing Systems Track on Datasets and Bench- marks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL httpps://daaasetts bewprassstteebbbbccnnm----eeeeee proceedinssoooeeiing.g.. neuriiinnnnnneuursppeeee. httppp.ccpppppppp poc/paper/2021/haat heeiaatsecobb///see2ccco1o1113f3 -awpaa/aaaaaa---ttttt---rrrooooo html
[29] Dong Huang. Jianbo Dai, Han Weng, Puzhen Wu, Yuhao Qing, Jie M. Zhang, Heming Cui, and Zhijiang Guo. SOAP: enhancing efficiency of generated code via self-opiimization. CoRR, abs/2405. 1 15899, 2024. doi: 10.48500ARXIV2205.55555
URL https://doi.orr ort///0i.org//00/
48550/arX 2405 15189
[30] Jie Huang and Kevin Chen-Chuas Chang. Towards reasoning in large language models: A survey. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 1049-1065. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023. FINDINGS- URI. http:::://///oo / 11.655/vv/202// findings-acl.cm
[31] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprini 20.0072300001900 2023.
[32] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample. Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Seao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06825,, 2023. doi: 10.45500ARXIV.2310.065
Utt.httss://ddo org/10. 48550/arX000 htt.://///..oolrg/////////////// 2310
06825
13

---

[33] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed, Mixtral of experts. CoRR, abs/2401 2024, doi: 10.48550/ARXIV. 3401.04885 URL attpp:///oii org/10 48550/arXiiy 2401 04088
[34] Jachun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations. In Yoav Goldberg, Zormitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 1266-1279. Association for Computational Linguistics, 2022. doi: 10. 10.11653/V1/200 222.22220.222.2221.
URL https
//doi.org// 100 18653//1/20000 2:06:///1/202....1mmmmm..m 82
[35] Daniel Kahneman. Thinking, fast and slow. Farrar, Straus and Giroux, 2011.
[36] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwa- sawa. Large language models are zero-shot reasoners. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neu- ral Information Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 Decem- ber 9, 2022, 2022. URL http::/papers......////// -tw.:eecc-.ot-----aaatt--------e
www.rccpppppppeeeeeeeeeeee files www.eefwwweeeeeeeeeeeeeeeaeeeeee 2aaper/2022/haae html
88b06291accd4ccf66e
[37] Rik Concel-Kedziorste Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, NAACL HLT 2016. The 2016 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 1 152-1157. The Association for Computational Linguistics, 2016. doi: 10. 10.18653/V1/NI6 136. URL URL http:/tttt///oiitii 1etp://.1111111//11 111131/61/111-..
10.
[38] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike. John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. CoRR. abs/2305. 200500 2023. doi: 10.48550ARXXV.2300 18550/arXi:0 2305. 20050
URL
[39] Ying Yi ai, 2024. URI httphttss::/ppatforo www.ieffrm..inyywwaaa.. wwnggyiinnnuuu com/
[40]
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Christian Bessicre, editor, Proceedings of the Twenty-Nint International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3622-3628. ijcai.org, 2020. doi: 10.2496//ICAA22220/50.11 org/ wttpg//....../////// ijcai. 2020/5011
URL https:
[41] Yinhong Liu, Zhijiang Guo, Tianya Liang, Ehsan Shareghi, Ivan Vulié, and Nigel Collier. Measuring, evaluating and improving logical consistency in large language models. arXiv preprint arXiv:2410.02200 2024.
[42] Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulić, Anna Korhonen, and Nigel Collier. Aligning with human judgement: The role of pairwise preference in large language model evaluators, 2024.
[43] Yiqi Liu, Nafise Sadat Moosavi, and Chenghua Lin. Lims as narcissistic evaluators: When ego inflates evaluation scores, 2024.
[44] Jianqiao Lu, Zhiyang Dou, Hongru Wang, Zeyu Cao, Jianbo Dai, Yingjia Wan, Yinya Huang, and Zhijiang Guo. Autocv: Empowering reasoning with automated process labeling via confidence variation. CoRR, abs/2405. 2024. doi: 10.48550/ARXIV.2405..
URI
attps://doo org/10. 48550/ arviv 2405. 16802
14

---

[45] Jianqiao Lu, Zhengying Liu, Yingjia Wan, Yinya Huang, Haiming Wang, Zhicheng Yang, Jing Tang, and Zhijiang Guo. Process-drive autoformalization in lean 4. CoRR, abs/2406.01940. 2024. doi: 10.4550/ARXIV224066000 01940
UtI.httss://doo httttp:://oo..or//10446666666663 w1w//10..465////11vv 2406
[46] Brian W. Matthews. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica el biophysica acta, 405 2:442-51, 1975. URL. httthhts:://ap semanticschol.at
44596673
[47] Meta. Introducing meta llama 3: The most capable openly available Ilm to date, 2024. URL https://ai meta. wtwcbloommmaeeocecccco mwtgg-/eeaaaaaa 11ama-33
[48]
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. LILA: A unified benchmark for mathematical reasoning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 5807-5832. Association for Computational Linguistics, 2022. doi: 10.18653/V 1/2022. EMNLP-MAIN.392. URL
1ww.3/16611/// v222222 emnlp- main. 392
[49] Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of GPT--4 CoRR, abs/2306.02707,, 2023. doi: 10-45550/ARXII22300022000
URL.https://ddo..rr
10. 485454855.aii... 2306. 02707
[50] OpenAI. GPT-3.5 Turbo, 2023. URL
gpt-3-5
[51] OpenAI. GPT-4 Technical Report. CoRR, abs/2303.08874, 2023. doi: 10.48550/arXiiv23003.
08774. URL https:///doi..rrg//0 2tw.:044450////.....0000 08774
[52] OpenAI. Introducing openai op-preview, 2024.
URL https://openai.com//nne
[53] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe, Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewriPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http:://p// //pppess nips cc/ wacppppp_____ files/ wateee//ppp////eee /002// A0t-800050001311-Aaraaatttt Abwtraat-t-taaaaeeeecceeec Conference. html
[54] Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations, 2024.
[55] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NI. models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilck HlakkaniTüit Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-ILT 2021, Online, June 6-11, 2021. pages 2080- 2094, Association for Computational Linguisties, 2021. doi: 20.18653/V1/2021..0. WAA.///222.AAAAAAAIIIIAACCC 168. URL http:http://alt org/10 10 18653/v1/
2021 www.naaalllammmm main. 168
[56] Archiki Prasad, Swarnadeep Saha, Xiang Zhou, and Mohit Bansal. Receval: Evaluating reasoning chains via correctness and informativeness. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6- 10, 2023, pages 10066-10066 Association for Computational I. Linguisties, 2023. doi: 10. (0.18653//1/2023 EEM.P....EPPPP.. MAIN.622.
URL TRLLhtps:: httph::t////e //doi. org/ 10. 18653/v1/2023 emnlp main. 622
15

---

[57] Shuofei Qiao. Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. Reasoning with language model prompting: A survey. In Anna Rogers, Jordan I.. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACI. 2023, Toronto, Canada, July 9-14, 2023, pages 5368 5393. Association for Computational Linguistics, 2023, doi: 10. 18.53//3/V ACL URL httLhhttss//ddt tttpp::/ddoioorg/00 18653/v1/2023. acl acl-oon... 294
[58] Abulhair Saparov and He He. Language models are greedy reasoners: A systematic for- mal analysis of Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
In The Eleventh International Conference on Learning
itttps://ppenreviee wttp:/ewwwee//dtfteeeeeeee
[59] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonoo, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexan- der W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew K. Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubara- jan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, and et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. CoRR, abs/2206. 2bbb//206004615, 2022. doi: 10.48550/ARXII.22066000 04611. URI. https://dois org/10.co 48550/ 11850//arivv 2206 04615
[60] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challeng- ing big-bench tasks and whether chain-of chain-of-thought can solve them. In Anna Rogers, Jordan L. Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 13003- 13051. Association for Computational Linguistics, 2023. doi: 11/1655//112023F11DDDDI ACL.824.. URL org/ 10. 18653/ v1/.023.2222222/ winninnnnnsaaae 824
https:///oot htt/ddi..ogg///
[61] Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. Proofwriter:
Generating implications, proofs, and abductive statements over natural language. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Findings of the Association for Computational Linguis- ties: ACI/JJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/UCNLPP 2021 of Findings of ACL, pages 3621-3634 Association for Computational Linguistics, 2021. doi: URL https://aoi.org//t 10655///110011153/////22222
10.18653/V1/2021.HBP FIN.22...FIIINGSACLL...
f findings-acl... .37
[62] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL- HLT 2019, Minneapolis, MN, USA. June 2-7, 2019, Volume 1 (Long and Short Papers), pages 41 4149 4158. Association for Computational Linguistics, 2019. doi: 10. J0.11855/V 1F559/11NN1911421 URL httph:tt::
org 10. 11653//1//1999 1421
[63] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint w:Xiv:2403.08295, 2024.
[64] Gladys Tyen, Hassan Mansoor, Peter Chen, Tony Mak, and Victor Carbune. Llms cannot find reasoning errors, but can correct them! CoRR, 0bss/2311.085/// 2023. doi: 10.48550/ARXIV. 2311.08516. https:/tt:///dd org/10 18550//rXiis www.aawww........... 2311. 08516
[65] Peter Cathcart Wason and Philip Nicholas Johnson-Laird. Psychology of reasoning: Structure and content, volume 86. Harvard University Press, 1972.
16

---

[66] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe Le, and Denny Zhou, Chain-of -hain-of-thought prompting elicits reasoning large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http: /ttp:p:ppppppe..eee http:///660611222ee1111111000100 Aat.::laaaba--attttt-t-------eoe Confeaerct--oofeeeceeeee htteererneeeeeemlll.......
[67] Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
net/pdf pidee2a3
[68] Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. Evaluating mathematical reasoning beyond accuracy. CoRR, abs/2404.05669, 2024. doi: 104855WAARXXV22244.006622
URL httph://pp/oo..... org/ 10. 488.50/5rraii.... 2004.444400 05692
[69]
Jing Xiong, Zixuan Li, Chuanyang Zheng, Zhijiang Guo, Yichun Yin, Enze Xie, Zhicheng Yang. Qingxing Cao, Haiming Wang, Xiongwei Han, Jing Tang, Chengming Li, and Xiaodan Liang. Dq-lore: Dual queries with low rank approximation re-ranking for in-context learning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [tRLLatts///ppeeeevveee htwp://wwwwww...eff.reeeeeee net/forum? id=qhoxvePSlq
[70] Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein, Re3: Generating longer stories with recursive reprompting and revision. CoRR, abs/2210.06774, 2022, doi: 10.48550/ARXIV. 2210.06774. URL https://doi org/10 10. 48550/arXiv. 06774
[71] Zonglin Yang. Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. Language models as inductive reasoners. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference the the European Chapter of the Association for Computational Linguistics, EACL 2024 - Volume I: Long Papers, St. Julian' 's. Malta, March 17-22, 2024, pages 209-225. Association for Computational Linguistics, 2024. URL org/ 2024. eacl-long. 13
[72] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http:://ppprrrs nips cc wwtpsspr.cppppppppee haapp_aflllee faaper_filessppper///000 pptppppppe/222//sa Conftet-ooooeooeoe hat.meeeenneeemmlllll html
[73] Yuxuan Yao, Han Wu, Zhijiang Guo, Biyan Zhou, Jiahui Gao, Sichun Luo, Hanxu Hou, Xiaojin Fu, and Linqi Song. Learning from correctness without prompting makes LLM efficient reasoner. CoRR, abs/2403. 19094, 2024. doi: 10.48500ARXX..240...000 URL https://áoi.org org/10, 48550/ arxiv. wwe.xi..203.... 19094
[74] Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H. Chi, and Denny Zhou. Large language models as analogical reasoners. CoRR, abs/2310.01714, 2023. doi: 10.48550/ARXIV.23100001 18550/arXiv.. 2310. 017114
URL potttps://di..ooe
[75] Xi and Greg Durrett. The unreliability of explanations in few-shot prompting for textual reasoning. In Sanmi Koyejo. S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http:: www..cc/ppppppppoo.. cc/ wwwpeeprppppppppepppcccceee....p hacplappperlelleeeel fiwes/ wwwle//ppppr//22222 /022/ hash/ hath://6000000011100222222252260
.ttll
[76] Wenhao Yu, Zhihan Zhang, Zhenwen Liang, Meng Jiang, and Ashish Sabharwal. Improving
language models via plug-and-play retrieval feedback. CoRR, abs/2305. 14002, 2023. doi:
10.48550/ARXIV.2200 14002. httpst s/stts::/doi.. orreii.rrg/000 /00 48550/ arrvv..iiii.v./iii 2305. 14002
17

---

[77] Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang, and Jiaya Jia. Mr-gsm8k: A meta- reasoning benchmark for large language model evaluation. CoRR, abs/2312. 17080, 2023. doi: 10:48500ARXI.2311. 17080. URI htt.t:ttp/s/.... wrg/10.../..0 400.48850/aaa arXiv 2312. 17080
[78] Chi Zhang, Baoxiong Jia, Mark Edmonds, Song-Chun Zhu, and Yixin Zhu. ACRE:
abstract causal reasoning beyond covariation. In IEEE Conference on Computer Vi- sion and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. pages 10643- 10653. Computer Vision Foundation IEEE, 2021. doi: 10.1109/CVPR46437.2222 01050. URL https::/
theovf. http://aataat//////22//////////a
ACRE Abstract__sccc.........a. Caacctl_Cauualla REasoningnnni..... www.oooigg.eeooodddooooooeoaaeee Covariation CVPR 2021 paper_ppper.re .ttll
[79] Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. In The Twelfth International Conference on Learning Representations, 2024. URL httpp:///ppnurevlew..e
[80] Yongehao Zhou, Andrei Toan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL htt.ttts///opeeeeeviiew wttperrwwiwwwweeeeeeee ?id 9d992gv822DE.
18

---

A
Appendix
A.1
Limitations
The meta-reasoning evaluation framework in MR-Ben, while innovative, is not without its limitations. Firstly, its applicability may be restricted when it comes to subjects that are inherently holistic or creative in nature, such as humanities or sociology. These subjects often require a comprehensive understanding and modification (e.g. essay writing), which can be challenging to break down into specific, sequential reasoning steps and corrections. Secondly, MR-Ben is currently confined to questions in English. This could potentially limit the scope of reasoning challenges that can be explored, as different languages may present unique cognitive and linguistic hurdles. Lastly, the analysis and correction of errors in the reasoning steps are currently based solutions generated by three LLMs, namely GPT-3.5, Mistral-Medium, and Claude 2. It's important to note that different LLMs and different individuals, may exhibit distinct reasoning and error patterns. Therefore, it would be beneficial to broaden the spectrum of solutions analyzed, incorporating more diverse range of LLMs and even human responses. This would not only enhance the robustness of the evaluation framework but also provide more nuanced understanding of the reasoning processes at play.
A.2 Broader Impact
Positive Societal Impacts The proposed dataset MR-Ben has the potential to bring about significant positive societal impacts. can contribute to the development and enhancement of L.MMs by providing comprehensive benchmark suite, which researchers and developers can the limitations and weaknesses of their models. This can lead to more accurate, efficient, and reliable LLMs. The meta-reasoning paradigm might open new avenues in Al research, leading deeper understanding of reasoning capabilities and the development of innovative methodologies for their evaluation and improvement. Moreover, with a wide range of subjects, MR-Ben can be a valuable resource for educational AI tools, providing personalized learning experiences and helping students understand and improve their reasoning skills. AI systems with improved reasoning capabilities can also be instrumental in various sectors, including healthcare, finance, and environmental management, aiding in complex cacision-maiing and problem- solving tasks.
identify and address
Negative Societal Impacts MR-Ben may also present potential negative societal impacts. As with any technology, there is a risk of LLMs being misused or used maliciously. For instance, LLMs with advanced reasoning capabilities could be used to manipulate information or deceive people. The of LLMs in decision-making making and problem-solving tasks could lead to over-reliance these systems, potentially undermining human judgment and critical thinking skills. Advanced LLMs, especially those used in sensitive sectors like healthcare and finance, need to handle vast amounts of data, which can raise privacy and security concerns if not managed properly.
A.3 Additional Related Work
Improving Reasoning Abilities of LLMs To enhance the reasoning capabilities of LLMs, prior research primarily focuses on specific prompting techniques [TT]. Existing efforts include few-shot zero-shot prompting with specific instructions 36 [4]. Although these methods have shown promising results, their
prompting with intermediate steps augmented demonstrations [66] 72 69
effectiveness is often constrained by their task-specific nature and the labour-intensee process of
designing prompts, leading to inconsistent outcomes across different tasks 75 8Q) Another strategy to facilitate reasoning involves instruction tuning or knowledge distillation, which elicits reasoning
paths from LLMs without explicit prompting [13] 49 26 44. These approaches typically involve resource-intensive fine-tuning over LLMs and require a large of examples annotated with COT.
Learning From Feedback Improving LLMs through learning from feedback has become preva- lent strategy, notably through reinforcement learning from human feedback, which seeks to align LLMs with human values by refining their outputs based on feedback [53] g However, this method faces challenges such as high costs due to manual labor and a lack of real-time feedback capabil- ities 20). An alternative strategy involves using self-correcting LLMss, which rely on automated feedback to iteratively adapt and understand the consequences of their actions without relying on hu-
19

---

Model
Coding
Phy.
Bio.
Math
Med.
Chem.
Logic
gpt-4-turbo
83/55
137/15
164/11
305/46
194/25
166/27
192/16
deepseek_come
100/38
145/7
167/8
321/30
200/19
172/21
193/15
Qwen2-72B
99/39
142/10
167/8
312/39
195/24
172/21
200/8
Table 5: Scoring of errr reasons from different models across subjects.
Coding
Phy.
Bio.
Med.
Chem.
Logic
Math
Agreement Ratio
7/8
12/13
21/21
12/12
15/17
15/16
10/13
Table 6: Agreement ratio between the author and the proxy scoring model across different subjects.
mans. This feedback can be derived from outside sources such as other models 70
45 tools 24
knowledge bases 21 76. evaluation metrics 34 67 or generation logits
B Robustness of MR-Score
Does the ACC_reCCsseeeeeeeeeeee reason metric's dependency on the judgments of different LLMs or human evaluators lead to variability in scoring ?
Question:
Answer: We would like argue that due to the careful design of our evaluation mechanism, the automatic scoring of error reasons both robust and economically feasible:
Multiple annotators: During the annotation stage, we collected multiple annotations for the first error reasons and potential error rectification from different annotators who agreed on the solution correctness and the first error step.
Proxy Model Evaluation: Based on the ground truth annotations collected from various perspectives, the proxy language model (e.g., GPT-4-Turbo) then examines the error reasons provided by evaluating models. Given the question/solution pair and information regarding the first eтrоr step, eπrоr reasons, and rectification, the potential flaws of the error reasons provided by the evaluating models are easy to diagnose under contrast.
ACC_reason reesoon robustness: Table-5 shows the scores of error reasons sampled from our evaluation results. For the same set of eгror reasons collected in each subject, three different models made their predictions on correctnessincorrcttess. We can clearly see the consis- tency of their predictions among the three models over questions in all subjects. Since the MR-Score is a weighted metric, the final score variability is less than percent in total.
Human-MMded Agreement Rate: As mentioned i the agreement rate between manual annotations and the GPT-4 predictions over 100 samples randomly collected from all subjects is 92%. Below is the exact detail of our setup:
We randomly collected 100 data instances where the evaluating model correctly identified the solution correctness and the first error step across all subjects. We then manually examined whether the proxy scoring model (e.g., GPT-4-Turbo-20244 -020--2024-04009) correctly scored the error reasons of the evaluating models. Table-Glis the detailed composition of the ratio in which the author agrees with the proxy scoring model. The annotation time varies significantly across subjects, as some problems-Soe as coding and chemistry- can take more than 10 minutes evaluate, while subjects like biology are easier to assess. This high agreement rate further supports the reliability of our evaluation, thus avoiding the need for manual annotation of potentially 138,000 problems (6,000 benchmark size times 23 models evaluated).
Question: Is the MR-Score sensitive to different weightings? Is MR-Score robust unified metric?
Table 7 shows breakdown performance for models all four metrics (MR-Score, MCC, ACC_step, and
1. Metric Robustness: Due to the progressive nature of the definitions of our subtasks (e.g.. the success of subsequent tasks depends on the previous ones), we can see the diminishing trend in the scores of MCC, ACC_stepp and ACC reason. However, thanks to the design
20

---

Table 7: Evaluation results breakdown MR-Ben: This table presents a detailed breakdown of each model's performance evaluated under metric ACCC/ACC-ste/ACCC--C. ACC-reson across different subjects. Here k stands for number of shot and every model used in this experiment are instruction-tunm
Model
Bio.
Phy.
Math
Chem.
Med.
Lugic
Coding
Avg.
k= A=1
k=U
k=
A=
A=1
k=
k=1
A=O
1=1
A=O
k=1
k=
k=1
k=
IR--Scores
Clande3-Haiku
5.7
5.8
3.1
3.1
6.5
1.2
1.2
4.4
3.1
3.6
6.6
5.7
6.7
5.7
5.4
4,9
6.7
3.6
4.4
1.7
4.5
3.0
4.1
4.0
5.5
PhiB-3.8B
13.4
12.5
12.7
10.8
13.3
13.1
16.4
17.1
10.2
8.1
8.4
5.3
9.1
10.2
11.9
11.0
B
74
5.5
7.2
8.6
78
6.0
5.5
4.6
6.7
7.0
6.3
Despsok-Code-778
9.9
11.
11.
11.9
10.4
11.0
9.8
LLaMA3-SB
12.0
11.9
10.9
7.5
15.0
9.0
12.6
12.7
9.3
8.0
9.4
9.6
15.8
10.0
12.2
9.8
Qwen1.S.72B
15.3
19.2
12,9
13.6
12.0
10.0
13.9
16.3
11.7
14.7
10.4
12.9
3,9
5,9
11.5
13.3
DeepSeek-67B
17.3
15.4
16.2
20.6
14.7
12.2
14.5
15.2
LLaMA3-70B
20.4
27.1
17.4
20.5
14.9
15.8
19.5
25.1
16.3
19.3
16.3
16.8
29,8
16.7
19.2
20.2
Mistral-
22.2
28.0
26.7
25.4
24.3
28.2
24.0
27.0
15.9
19.3
14.7
17.1
21.1
21.4
21.3
23.8
DepSeet-V--236
30.0
37.1
32.2
36.5
32.2
30.0
32.5
35.4
26.5
32.4
23.6
27.4
34.2
27.1
30.2
32.3
GPT-4-Turbo
47.3
42.8
45.2
44.3
44.0
46.0
38.8
38.4
34.1
33.6
53.6
57.3
43.2
44.7
MCC-Matthew Correlation Coefficient
Chaude3-Hiku
13.96
17.72
16:47
13:62
15,09
10.74
16.54
19.96
8.52
8.35
6.21
4.94
4.36
11.59
10.76
10.72
19.44
16.66
21.33
17.48 17:45
18.24
12.6
11.19
13.28
4,07
0
12.35
12.35
12.96
13.78
Dodpsck-Coder-35B
7.51
8.57
6.81 11.73
9.69 21.06
9.98
7.94
1.62
6.28
26.18
15:44
9.53
9.44
Deepseek-Codee Coder-
4.96
9.79
8:77
6.72
9.05 10.82
10.49
9.39
5.02
3.17
3.22
2.58
10.91
6.27
7.49
LJaMA3-8B
19.37
21.15
16.24
18.64
26.55
21.87
25.99
28.6
14.92
18:95
11.8
16.24
14.54
15.72
18.49
20.17
Phi3-3.8B
27.66
28.48
21.61
21.44
22.29 25.17
30.92
33.37
17.36
14.9
13.03
9.56
18.76 14.48
21.05 21.67
Qwen1.5-72B
33.64 42.44
31.4 31.56
29.2 23.28
36.47
29.64
24.42
27.74
13.8 15.69
29.55
66epseek-67B
43.61 41.73
24.16
28.77
24.95 23.87
36.58
37.29
27.8 28.93
26.74
25.09
28.23
29.06
30,3 30.68
LlaMA3-708
45.67 56.14
40,34
41.3
32.76 30.94
41.72
52.12
33.18
37.75
32.0 33.87
29.67 47.86
39.08
40.26
Mistral-Large
41.67
49.0
34.24
33.47
29.0 37.05
41.99
47.07
23.76
32.05
25.66 33.25
37.05 33.52
33.34 37.92
Despscek-v2-2368
52.96
53.38
46.48
43.75
40.53
54,32 50.15
37.61 44.53
36.36
35.41
35.7 45.89
44.67 43.74
GPT-4-Turbo
63.33
62.59
52.9
52.7
50.67
52.84
53.05
54.59
56.79 54,66
40.95
42.94
52.5
57.53
52.88 53.98
Avcuracy of First Error Step
Claude3-Haiku
2.15
3.1
1.4
1.12
2,38
1.59
1.77
4.42
1.69
0.68
1.01
0.29
0.0
0.0
1.49
1.6
GPT-3.5-Turbo
2.86
4.53
4,2
4.76
4.37
3.84
2.87
8.17
2.37
3.05
1.73
7.63
0.61
2.44
2.72
4.92
Deepseek 3oBs
14.83
10.29
14:94
12.36
14.69
10:92
15.67
16.31
14.54
12.43
12.22
18.18
5.49
3.05
13.2
11.93
Deepseek
Ceepeekk-Code-77 Coder-BB
7B
21.77
18.18
23.28
19.83
23.41
20.03
24.46
23.18
23.29
26.09
20,03
24.72
4.27
6.1
20.07
19.73
LlaMA3-8B
14.35
14.35
8.62 17.53
20.29
7.8
11.59 14.16
13:13
8.41
13.64
11.36
17.68
9.76
15.83 10.27
PhiB-3.8B
12.68
11.48
17.69
16.12
8.76
10.23
6.96
8.54
9.15
13.76
11.67
Qwen1.5-72B
11.48 15.31
10.63
11.49
10.79
9.88
14.38 12.45
11:03
13.49
8.1 10.94
1.83
4.27
9.47
11.39
Deepsock-67B
13.16 19.14
19.25
21.84
20.81 22.11
23.39 17.17
14.71
12.08
12.78
13.49
12.2
14.02
15.72
18.01
LlaMA3-70B
15.79
22.25
14.66
18.39
13.65 15.08
17.81
22.32
14.36 16.64
14.91
14.91
26.83
13:41
16.86
17.57
Mistral-Large
18.38
25.54
26.33
28.29
27.28 33.25
27.59
19.29
14.53 14.96
19.51
21.29
24.06
Deepseek-v2-2269
27.51
35.41
37.64
40.23
36.28 34.33
33.91
37.55
27.5 32,57
22,87
27.56
33.54
26.83
31.32
33.5
GPT-4-Turbo
41.77
46.06
42.58
46.5
46.49
47.81
42.6
46.8
37.06 36.72
29.93
33.24
50.61
59.15
41.58
45.18
Accuracy of First Error Reason
Clande3-Haiku
0.56
0.84
0.22
2.21
1.18
GPT.3.5-Turbo
1.19
2.63
2.24
1.96
1.85
1.46
0.88
3.53
1.35
1.69
0.72
4.17
0.61
1.83
1,26
2.47
Deepseek-Cooo Coder 33B
2.87
1.44
2.01
1.15
1.69
2.21
2.15
1.93
2.63
1.05
1.85
2.56
3.05
1.83
2.32
1.74
Ceepseek-Coder::20 7B
5.98
5.02
6.03
4.6
5.98
7.93
5.79
6.22
4.9
5.08
6.25
5.54
3.05
5.49
5.43
5.7
LlaMA3-8B
6.7
4.89
7.08
4.73
7.93
Phi3-3,8B
8.13
6.7
6.9
5.75
7.02
6.37
9.66
10.52
5.78
5.08
5.4
2.7
7.32
7.32
7.17
6.35
Qwen1.5-72B
10.29
12.2
6.9
7.76
5.85
4.81
6.22
9.44
8.06
9.46
6.11
8.24
1.22
3.05
6.38
7,85
Deepseek-67B
8.85
11.24
8.62
10.06
8.32
9.49
7.73
12.23
9.46
5.6
10.37
10.37
10.37
8.88
9.91
LlaMA3-70B
13.16
18.42
9,77
8.58
10.27
11.59
15.88
13.49
12.73
13.72
Mistral-L argee
15:27
21.0
19:05
20.45
15.1 21.72
16.56
18.54
13:03
14.21
11.22
11.94
17.07
17.68
15.33 17.94
Deepseck
22.25
31.58
25.57
30.17
25.1 23:28
22.96
28.11
21.54
27.5
18.89
24.15
29.88
23.78
23.74 26.94
GPT-4-Turbo
39.14
38.38
41.06
36.64
32.83
32.83
27.63
30.07
50.61
56.1
37.95
40.81
of our evaluation mechanism and metrics, the score rankings of different models stay in relatively stable order across metrics. In other words, we have not observed any model that excels in determining the solution correctness (thus high in MCC) but is unable to explain the rationale behind (c.g.. low in (eC_oon))))...
2. Task Difficulties: As shown the breakdown table, the ACC_ metric is more discriminative than the MCC metric for competent models but vice versa for the less competent ones. This aligns with our intuition that generally more difficult questions are more discriminative for strong candidates, while weaker ones are simply incapable of solving
21

---

Logic
Coreetrens: EmorRouone Recilled Steplaoon
PAYSSO
Step
LOGIC
Steppe
MEDICINE
Fire Frmm Eriee Rooson Aihh Reet fied Step
Step
MATH
AI
CHEMISTRY
BIOLOGY
CODING
Calling
Firz Erar
trr mussrr....
Roct Fied Stap....r
Figure 6: This the illustration of the dataset creation pipeline of MR-Ben We first compile set of questions from different subjects and then collect solutions from different LLMs. For each subject, a group of domain experts is recruited to annotate each question solution pair on its solution correctness, first error step, and error reasons.
them. This phenomenon could in part explain why in general the MR-Score is not very sensitive to minor changes in the weightings assigned to the subtasks, since the differentiability of the subtask metrics tends reconcile with each other under different scenarios.
3. Differentiability and Interpretability: The weights of the MR-Score are ultimately decided by considering both the discriminative ability and the interpretability. To best differentiate models with different evaluation results, we conducted a thorough grid search to investigate the impact of the weightings. Since the weightings calculated returned few optimal instances, we deliberately selected the one that assigns higher scores more difficult tasks. We believe the current weighting ratio strikes good balance between interpretability and differentiaton:: For example, GPT-4-Turbo, Deepseek-v2-2333 and Mistral-Large achieve 86.4% 78.5%, and 81.2 respectively MMLU but score 43.2%, 29.4%. and 21.3% in benchmark.
C More Discussion on Biases
To quantitatively assess the relationship between the length solutions and their correctness, Pearson- Airelatioo-Coefffcceen were calculated and reported in Table-Biin in the Appendix. The result suggests varying dynamics across disciplines regarding how solution length impacts the likelihood of cor- rectness. For subjects such as coding, chemistry and math, longer solutions are less likely to be correct, which could suggest that complexity or elaboration in responses may lead to mistakes or incorrect reasoning. For medicine, despite being weak, there's 's a tendency for longer solutions to be slightly more correct, possibly due to more detailed or thorough explanations being favorable. For the other subjects, length of solution does not appear to significantly affect correctness, indicating that other factors likely play a more dominant role in determining solution quality. The overall Pearson Coefficients analysis reflects the distinct nature of problem-solvingg in each field of our benchmark.
Table 8: Pearson Correlation Between Solution Length and Correctness
Subject
Pearson Correlation
P-value
Medicine
0.094
0.0072
Physics
-0.061
0.111
Biology
0.009
0.783
Chemistry
-0.127
0.00018
Coding
-0.199
0.0021
Logic
0.0002
0.995
Math
-0.115
0.00049
22

---

Table 9: Evaluation Results of Models on MR-Bean in few-shot settings: This table presents
detailed breakdown of each model's performance evaluated under metric MR-Score across different subjects.
Model
k-shot
Bio.
Phy.
Math
Chem.
Med.
Logic
Coding
Avg.
0
0.1
0.0
0.0
0.1
0.0
0.0
0.7
0.1
Gemma-28
0.0
0.0
1.0
0.0
0.4
0.2
0.0
0.2
0.1
0.2
0.7
0.6
0.7
0.2
0.0
0.4
0.1
0.3
0.7
0.3
0.0
0.4
11.1
14.9
14.8
12.8
9.4
9.6
9.1
11.7
Llama3-8B
11.7
12.8
7.3
5.7
9.2
9.7
7.8
11.1
8.8
6.4
6.2
2.4
7.5
10.0
10.7
8.3
8.2
5.5
5.3
3.0
7.3
19.9
15.4
15.0
17.6
14.6
13.5
28.2
17.7
Llama3-70B
30.5
21.4
16.8
26.2
16.9
16.0
15.3
20.4
27.2
19.9
16.8
22.0
15.9
17.5
19.5
19.8
27.2
20.6
16.3
21.1
16.0
14.6
19.4
19.3
44.7
42.8
44.3
44,0
38.8
34.1
53.6
43,2
GPT-4-Turbo
47.3
45.2
45.4
46.0
38.4
33.6
57.3
44.7
46.6
42.7
44.9
43.3
42.1
35.9
53.0
44.0
44.8
46.5
44.4
41.2
33.7
56.6
44.5
D Evaluation Prompt
Figure-It I0 is the prompt template we used to evaluate all the models our paper. Note that with minor modifications the following template, the evaluation results can be heavily affected. For example, by introducing a simple hint sentence "Hint: This solution is incorrect. Please focus on looking for the First Error Step and Error Reason.", ", the model performance
drastically improve as shown in 3 Also, by simply taking away the line 'Solution Analysis' in the response format part of the prompt. the evaluated model will directly output the scoring result without step-wise COT analysis on the solution. This setup will lead to a near zero MR-Score performance as discussed in Section-5
Figure-7 7 is the prompt we used to query language models for solution generation during the dataset compilation phase. Note that in the prompt, we specifically asked the model to analyse each option in the multiple-choice problem. This is crucial in examining the model possesses a comprehensive understanding on the topics that the question is asking.
Figure I shows the prompt we used to query GPT-4 score the error reasons returned from evaluated models. Despite the challenging nature of the original task to determine the solution correctness, is much easier job determine if the error reason from the evaluated models aligns with the ground truth error reason.
Figure-B demonstrates the prompt template used for self-refine experiment. Note that followed the setting of (31) without introducing any prior assumptions or knowledge. This minimum version of extra prompting would mostly rely on the capability of language models perform self-refine procedure.
E
Self Refine Analysis
In this section, we present the results of self-refine in the task level. Specifically, we are looking at the change of labelling by the evaluated models the determination of solution correctness as shown by Figure-D) We summarize our observation below:
Small Models like Gemma-2B are too limited to perform effective self-reflection.
Competent Models like GPT4-Turbo are confident in their initial decisions, hardly switch- ing their decisions during self-reflection.
Intermediate Models like Llama3-70B exhibit substantial changes during self-refiection, indicating lack of consistency their decisions. However, its change of decisions from incorrect to correct happens to be significantly higher in locating the first error step than
23

---

Prompt for Solution Generation During Dataset Compilation
Please generate step-by-step analysis for the following Question in the subject (subject).
Question: DnfFiloc[ion [
Question? }}
Choice_At cefffllo[idd [dx]]
Choice_
Choice_B: [df. iooiclooccooo idx]
Choice_B BB
Choice_C [df. (tw.1loo[iii idx
Choice_ [ooicce_c]] []]]
Choice_Dr D: (tf.:loo[[idx] [ Choice_o"""" [']}
Here is the desired format, please analyse each candidate choice sequentially then jointly decide which option is the solution in the final step.
Please ensure every newline character follows step indicator:
Step [The first reasoning of the step by step analysis the candidate
choices here]
Step 2: [The second reasoning of the step by step analysis on the candidate choices here]
... Step n: [Conclude your analysis and decide which choice to make here] Solution: Choice ChD/CD
Please follow this format without any additional introductory or concluding statements.
This is the prompt used for solution generation during the dataset compilation stage. Note that besides coding, every subject question in our dataset takes the form of multiple choice problem.
Figure 7:
in examining solution correctness and explaining the error reason, therefore boosting the overall MR-Score by a large margin. We believe the lack of consistency does not necessarily indicate a more robust or advanced reasoning ability, despite the increase in the evaluation results.
Conclusion: Our results support the observation that LLMs generally lack effective self- refinement capabilities (3I.
F Error Analysis
We provide qualitative analyses of how GPT-4 as an example model performed on our benchmark across all seven subjects. The purpose is to offer a deeper understanding of the types and causes of errors made by experimented models to inform future improvements. For each subject in the subsections below, failure case and success case are listed. Following the MR-Ben evaluation framework, each case demonstration consists of the following parts: (1) original questions, ground-truth final answers, and L.L--generted CoT solutions; (2) human annotations of wise error detection, explanation, and correction; (3) evaluation annotation from the experimented GPT-4 on the aforementioned LLM -LM-generated CoT solutions; (4) scoring results of the error reason if the experimented model identifies the correct first error step.
options,
From our analysis of sampled failure cases, several general observations are made. Firstly, the assessed model GPT-4 exhibits widely resistant 'false positive bias" on our benchmark across all subjects: In cases where the LLM makes incorrect evaluations, the proportion of type I errors is much higher than type errors. In other words, GPT-4 tends to overlook the mistakes that exist in incorrect model solutions and mislabel them as correct, while seldom actively mislabeling correct model solution steps as incorrect steps. fact, among the 42 sampled cases we surveyed spanning the seven subjects, all failure cases (size 21) belong the type error category. We provide two

---

possible explanations for such bias: (a) input bias: the implemented LLMs are instructed-tume and are inherently biased to follow the prompt ipput. Therefore, even when the models are asked to fairly judge these CoT solution steps in the prompt input in binary labels, it is likely their labeling threshold is affected and biased towards positive judgments. This is common issue using LLMSs as generation evaluators and may be mitigated by adjusting the prompt design or other debiasing methods 79]; (b) self-preference bias: it has drawn recent attention that state-of-the-at models display celf-prffeeenc bias: the phenomenon in which an LLM inherently favors their own generated output over texts from other LLMs and humans EK] 54]. Therefore, the experiment results of LLMs that are under the same family of the three sampled models (GPT-3.5-Turbo-011
[50], Claude2 [3), and Mistral-Medium (32) may be affected. With the increasingly extensive use of self-evaluation and LLM-as-jude methods, we call for future researchers' attention to the potential issue.
Secondly, the MR-Ben benchmark revealed many intricate cases where the assessed model GPT-4 reached a correct final answer through incorrect solution steps, challenging the models' multi-step reasoning capabilities to a greater scale. As shown in the failure cases in math, physics, biology, etc., our benchmark evaluation is able to identify step errors that the sample model made in the solution steps even when its final answer matches the final ground-truth choice. While such step errors can be trivial in terms of generating the correct final answer in the demonstrated failure cases, they can become significant in just slightly nuanced questions, as mentioned in the error analysis section of framework, by decomposing the question and model solutions, remains
MMLU [27]. In contrast, relatively immune to the nuances in question framing. This highlights an important significance of our MR-Ben benchmark in that it is not only elaborate but also robust compared to previous benchmarks.
Lastly, there are subtle nuances of model performance different reasoning paradigms manifest in the case demonstrations of specific subjects. They are interpreted case by case by the captioned figures listed below.
25

---

E.1 Mathematics
Subject: High School Math
Question: What is the smallest whole number that has remainder when divided by 4, a re- mainder when divided by 3, and remainder of 2 when divided by 5?
Options: [A.] 67 [B.] 43 [C.] 57 [D.] 37
Ground Truth Answer: D
Sampled CoT
Sampled Model: Claude2
Model Solution Steps:
Step 1: Check Choice A (67) When 67 is divided by 4, the remainder is not 1. So 67 is not the correct answer.
Step 2: Check Choice B (43) When 43 is divided by the remainder When 43 divided 3, the remainder is 1. When 43 divided by 5, the remainder 3, not 2. So 43 is not the correct answer. When 57 divided 3, Step 3: Check Choice C (57) When 57 is divided by the remainder
the remainder O, not 1. So is not correct answer.
Step 4: Check Choice D (37) When 37 divided by 4, the remainder the remainder
When 37 divided by 3,
When 37 is divided by 5, the remainder is
Step 5: Choice (37) satisfies the requirements and is the correct answer.
Solution: Choice
Solution: Choice D
Human Annotation
Model Solution Correctness: incorrect
Model Solution First. Error Step: Step
Model Solution Error Reason:
The model's calculation is incorrect. In step the model incorrectly calculated the remainder of 43 divided by 4 as 1 instead of 3. The model assumes that when 43 is divided by 4, the remainder is 1, but it should be 3. There is miscalculation of the remainder here.
Model Solution Rectified First Error Step: Step 2: Check option (43): When 43 divided by the remainder 5, the remainder 3, not 2. Therefore, 43 not the
When 43 divided 3, the remainder When 43 divided by answer.
-Mdee Evaluation Result-
Evaluated Model: gpt-4-turbo
Evaluation Raw Response: Solution Analysis: Step 1: The step checks remainder when 67 is divided by and finds that it yields a remainder of 3, not 1. This a correct verification hased on the criteria specified in the question.
Step 2: Checks the remainder of 43 when divided by 4, 3, and 5. The findings (1, 1, and 3 respec- tively) are verified correctly and match the expected remainders except for the one with 5, correctly identifying that the condition is not met for division by 5.
Step 3: The step calculates the remainders for 57 being divided by 4 and 3. The remainder when divided by 3 does not match the specified conditions (expected was 1, result was 0). Hence, the claim that 57 isn't 't correct is valid.
Step and Step 5: Computes and verifies the remainders for 37 when divided by 4, 3, and 6, matching all the expected remainders accurately, and concludes that 37 meets all conditions specified in the problem.
The selection of Choice D for 37 is based accurate verification that fulfills the problem's constraints.
Solution Correctness: correct
First Error Step: N/A
Error Reason: N/A
Error Reason Scoring
Need Error Reason Review: False
Figure E.1-1. A failure case in the math subject. This is a question where the sampled LLM (Claude2) reached a correct answer through incorrect multi-step reasoning. In 'Evaluation Results', the model fails to identify an error in step 2 of model solutions. As noted by the human-annotated 'Model Solution Error Reason', Step checks the remainder when 43 divided by 4 and incorrectly calculates the remainder as while the correct calculation 3. The failure case showcases one of the strengths of our benchmark in probing the intricacies of multi-step reasoning by examining process-level errors.
21

---

Subject: high school math
Question: Alex needs to borrow $10, 000 from the bank. The bank gives him two options. 1. A ten-year loan with an annual interest rate of 10% compounded quarterly, with the condition that the end of years, Alex must make payment equal to half of what he owes. The other half continues to accrue interest, and at the
ten years, Alex will pay annual interest rate of 12%. with just one lump-sum payment at the end of the ten years. Find the positive difference between the total amouuts Alex has to pay back under the two schemes, Round your
remaining balanoe. 2. ten-year
with simple
to the nearest dollar.
Options: [A.] 16401 [B.] 16400 [C.] 382 [D.] 383
Ground Truth Anawer:
Sampled CoT
Sampled Model: GPT3.5
Model Solution Steps:
Step 1: Calculate the total amount Alex has to pay back under option 1.
Step Calculate the total amount Alex has to pay back under option 2.
Step 3: Find the positive difference wetween the total amounts Alex has to pay back under the two schemes, step with the given choices.
Step Compare the positive difference calculsted
Step 5: Round the calculated positive difference to the nearest dollar if necessary.
Step 6: Determine which choice matches the calculated positive difference. Solution: Choice A
Solution Correctness: incorrect
Model Solution First Error Step: 6
Model Solution Error Reason: Acrording the logic provired earlier, the model should give correct. match in the end, but the final answer given is wrong, In this step, the model selected the wrong answer, which could be due errors in the deduction process that were listed.
Model Solution Rectifled First Error Step: Step 6: Determine which option matches the calculated positive difference, Answer: Option D.
-Model Evaluation Result-
Evaluated Model: gpt-4-turbo
Evaluation Raw Response:
Solution Analysis:
Step Neutral- This step claims calculate the total amount Alex has to pay back under option but actual calculations or mathematical process.
does not provide
Step 2: Neutral Similar to step 1, this step lacks any substantial content regarding how calculations are carried out for the repayment under option 2.
Step 3: Nentral This step mentions finding the positive difference between the total amomnts though no shown.
specific values calculations
Step 1: Neutral - Comparing the positive difference to given choices depends on values obtained in previous steps, which are not provided.
Step 5: Neutral The mention of rounding the difference falls in line with the problem's 's Instructions but is non-specific due lack of previous calculations.
Step 6: Incorrect- Determining the choice without showing calculations leads to unsupported conclusion. Solution Correctness: incorrect
First Error Step: Step 6
Error Reason: The first error step is Step 6 where a decision is made on which choice matches the calculated
positive difference without providing any calculations or reasons for the decision. The rectified reasoning should involve calculating the payback amounts for both options explicitly, finding their difference, rounding it if necussary, and then correctly selecting the matching option based on those calculated and verified figures.
Need Error Reason Review: True
EEror Reason Scoring
Error Resson Correctness Analyais: The incorrect solution directly jumpe the finsl choice without showing any calculations or logical reasoning that leads to that decision. This skips essential steps that would pormally validate the conclusion drawn. The ground truth error reasons indicate that the error lies in the final selection of the answer without considering the calculation errors logical missteps in previous steps that may have led to an incorrect conchusion. Essentially, the error is not in the methudology but in the final stop's exccution, where an incorrect choice selected without showing the work that would justify this selection.
Error Resson Analysis: The student correctly identifies that the error occurs in the final step, where decision is made without showing calculations or reasoning. This observation aligus with the ground truth that the final step incorrectly concludes without apparent justification. - The student emphaeizes the need for explicit calculations and clear demonstration of the reasoning process that leads to the selection of the correct option. This suggestion correctly identifies the lack of shown work and logical deduction the provided incorrect solution. The student's explanation focuses the absence of explicit calculations and reasoned declalon-making in the final step, which accurately reflects the actual error in the solution. Error Reason Correctness: Correct
Figure 1-1 success case in the math subject. The assessed model identifies the key
mistake in Step 4 regarding the faulty equation f(8) (10) = f and points out
the incorrect and bascless cancellation leading to f(8) = f().. Its crror reason also
correctly suggests that the proper method involves using the transformation properties
iteratively in steps of 2 units backward from f10) to reach f0).
22

---

E.2 Physics
Subject; collegs.phystes
Quextion: Protons used in cander therapy are sypleally accslerated to about 0.5c. How much work must be done on particla of moss mn in order for is to reach this spced, asouming is starts as reat?
Options: [A.] 0.25me² B.] 0.60me2 [C.] 0.87me2 [D.] 1.25mc2
Ground Truth Answer: A
Sampled CoT
Sampled Model:
Clande2
Model Solution Steps:
Step 1:
The workseuergy sheotemm states that the nes work done on an objeet equels it change in kiuotie energy.
Step 2:
The kinetic energy of an object with masa moving speed is
energy 2 mmv?.
Step Since proton
Step final speed
Step Plogging laso the kinetic energy equation gives!
Final - [m(066))
Step B: Setting the change in kinetic energy egual to the not work done:
Net work Finul Initial KE- _m(00.c)² 0.36mc²
Step 7: Of the cholcas given, 0.36me2 is closest to 0.25m
Solution: Choice A
Solution Correctness: incorrect
Model Solution First Ervor Step:
clussical kinctic eneray formula KE } mvv culculate kinetic energy of particls moving at speads clese to the speed light. When the particle's speed approacher the speed of light, the relativistio kinesic energy formula KE = (t 1ime? should be used, where in is the rolativistic mass of the partielee ma is the rest mase of the partieln, and o Is the spand of light.
Model Solution Error
This incorreesly
In physics, when the velocity of an abject apptouches the apeed of lighi, we need to we the principles of special relativity to caleulate. la shis ease, the xinctic caergy of the particle is no longer the clussial kinetic caergy foruula but requires the use the relativistic kinetic energy formula, Model Solution Rectifled First Error Step: Step 2: The relativiatic kinatic energy of an object with mase and velucity
Step 2: The kinetic of an objact with mase moving valocity mc2
Model Evaluation Result-
Evaluated Model: gpt-d-turbo
Evaluation Raw Response:
Solation Anslyaia:
Step 1: Correct, The work-snargy theoram properly states that the not work donn on an objjct is equal to ihe change in
kinetic energy
Step 2: Incorrect. This is where the solution maker a fondamantal error by using the classical formula for kinetic energy which fen's valid at nigaifieant (cactions of the speed of light (like 0 Ge),
Step 3: Correct. The Inisial kinesc energy being zeto since the partiele starts from rect is a valid statement.
Step Correct. The final speed the proton la appropriately kiven as 0.6c.
Step 6: Incorreci consequence Step 2, applying the incorrect equation.
Step Incorrect
follows from the incorrect application of the kinetic energy formala and enda up with wroog capres-
Step week
ressoning comparing wrongly calenlated
wrong
Solution Correctnese: incorrecs First Breor Stepe Step Brror Reason: Stop 2 the non-relativistio formula for kinetic enerky. 2 mop? 2, which nos accurate for speeds approuching the speed of light (auch as 0.6c). The zelativiatic kinctic energy factor, 9
should be instead, which (v 12me?
y le
Usiing formola
correctly accounts
relativistic effects significunt at high velocitios.
Step: Step
Eeror Rensoni Step 2 uses the nowerelativistic formula for kinettc energy, 2 2 which not accurate for speeds appronching speed light (such as 0.6c). The relativistic kinctic eaergy formula should be mnd iastead, which Kg (7 1)me² where v ls the Larentx factor,
Using this formula correctly accomnts for the relativissle offects eigntficant. as
high volocitica. Need Ersor Roason Reviow: True
Brror Reason Corractness Analysle: - As spande appreaching that of light, relaciviatic offects become significant. Thus, the classical mechanies approach ts no longer sccurate for describing the physics of the sitmation. The porrect formula to calculase kinesic energy al relativistic speeds includes the Loreats factor, Ti which accouate for the offeets of time dilation and length contraction high velocitioa. The relativiatic kinetic energy (KE) formula in KE ( 1)mge?, where mp the Loreula fuctor, defined as 5
rest mass the particle, e ls the speed of light, and
This formula divergeo
from the classical RB 1 2 when becomes eignificant fraction of Therefore, the Incorrect step the solution wing non-rclasiviatic kinetic energy formula, which fail account for the relosiviatic eifccts that are siguificunt
the The onergy formula, which includes the Loventa
Analysie:
correctly identifies
enargy formals (5). The student provides correct yelativistic kinetio formola Overall, the student's explanation properly reflects the nature
non-relativiatic
KB = (r- - 1)me? and diserhhe y correctly as 7
of the ozrer in the solution and identifies the apptopriate zelativistie formula to use. Error Reason Correctness: Correct
Figure E.2-1. A failure case in the physics subject. It is another occurrence where the sampled LLM (Claude2) reached a correct answer even though its model solution contains an erroneous step. Physics formulas require strictly case-sensitive expressions to represent different concepts, as noted in 'Model Solution Error Reason', which models did not differentiate, thus making mistakes in explaining the relationship between the orbital period T and the satellite mass m.
23

---

Subject: collegs.physiee
Question: Protons used in canser therapy are sypieally accelerated to about 0.8cc How much work must be done on particln of mass ma in order for is to reach this speed, assuming starte as rest?!
Optious: [A.] 0.25mmc B.] 0.60me?? [.] 0.0Tme? [D.] 1.25me²
Ground Truth Auswer: A
Sempled CoT
Sampled Model:
Claude2
Medel Solution Stepa:
Step 1: The worksenergy theorem states that the net work done on an object equals ite change in kinotle energy.
Siey 2:
The kinctic euergy of un object with masa mn moving speed ia gwv...
Step Since the
kinetic energy
final
Step Phagging this lnto the kinetic energy equation
Final - mm0.6c))
Step Bi Setting the change in kinetic anergy equal to the not work done:
Net work Final Initial KE m0) 0.36mcc"
Step 7: Of the choicns given, 0.30mme2 in closest to 0.25me22, Solution: Choicn A
Solution Correctnesa: incorrect
Model Solution First Error Step:
Model
step incoreetly
energy
K 3 mu² calenlate
kinetic energy of particls moving at sponds close to the speed light. When particla's speed approaches the speed light, the relativiatic kinetic energy formuls KR (t 1ime? should be used, where in is the relativietic mass of the particll,
my is the rest masz of the partleln, and ls the spand of light.
Model Solution Rectified First Brror Step: Step 2: The kinetic anergy of an objact with mase in moving at velocity in
Model Evalustion Result-
Evaluated Model gpt-4-turbo
Evaluation Raw Responsei
Solation Analysia:
Step Correct, The work-seery theerem properly states that net work done on an object is equal its change in energy the particle from is valid statement. 0.6o.
kinctic eneray.
by ueing clasetcal
speed light
Correct. The inisial kinesic being
Correct, The final speed conequeece Step 3, applying the follows from the incotrect application of the kinetic
is
equation. formula and enda with Stepp 7: Incorecci rensoning comparing the wrougly calculated work with the auswer cloicen, londing to the wrong selection.
Incorrect
Step 6: Incorrect
sion for ucl woek done.
Solution, Corgectness: incorrect First Breor Step: Step 2 Ereor Reason: Step 2 usee the non-relativintic formuls for kinatic accurate for speede approaching the speed of light (ench 0.6c). The relativietic kinetic Using this formola
energy: 2 mu? which
formola should be used instead, which is KB (v 1)me? where y la the Lorentx factor,
correctly accounts for the relativistic effects signifiennt at high velocitioa.
Solution Correctneas: incorrect
First Error Step: Step
Eeror Renson the speed of light (euch 0.5c). The relativiatic kinetic formula should be ueed instead, which ICB (7 1)me? Using formula correctly accounte for the celativistic effects signifieant as
the non-relativistic formuln kiuetic euergy, i mnv", which
speeda appronching
where y the Lorents factor, 7
high valocitios. Need Error Resson Review: True
Error Reason Correctness Analysia: speeda appreaching that of light, relasiviatic effects become sixnificant. Thus calenlata anargy
approsch speede
longer
physics
length contraction at high velocities. The relativistic kinetic energy (KB) formula is K (1 1)mgo?, where ma is the Thii formels divergee
the particla, ie the speed of light, and the Lorentx factor, dafined
from the clusical KE f mmm when e becomes a signifirant fraction of c. Therefore. the incorrnct step in the solution winn the non-reinsiviatic kinetic velocity of 0.6s.
formula, which fails account for the relativiatic effects that are siguificant
Eeror Reuson Anelysis: The student correclly ideatifics that the non-relativiatic kinctic energy Jormula is insecurate for spenda approsching the speed of light. - The studens accurately noter the requirement to use the relativistic kinatic energy formula, which includes the Larents factor (5). The student provider the correct ralativietic kinatic maergy formula 1)me? and discribes correctly
KB
Overall, the studont's explanation properly reflecta
/1
the Eevor Reason Correctness: Correct
solution and idantifies appropriate zelativiatic formala
Figure E.2-2. success case in the physics subject. The evaluator GPT-4 correctly identified the first error step and provided appropriate and rational reasons.
24

---

E.3 Biology
Subject: boologe
Question: Which of the following statements most detracts from the claim that the human appendix
a completely vestigial organ?
Options:
-[A.] The appendix can be surgically removed with no immediate ill effects.
[B] The appendix might have been larger in fossil hominids.
-CC.] The appendix has a substantial amount of defensive lymphatic tissue.
-[D.] Individuals with a larger-than-avraage appendix leave fewer offspring than those with a below-
average-sized appendix.
Ground Truth Answer: C
Sampled CoT
Sampled Model:
Claude2
Model Solution Steps:
Step I: Choice A states that the appendix can be surgically removed without ill effects. This suggests the appendix may not serve important function, supporting the claim that it is vestigial.
Step 2: Choice states the appendix may have been larger in fossil hominids. This suggests the appendix may have served more important function in our evolutionary ancestors, weakening the claim that it vestigial.
Step 3: Choice C states the appendix has substantial lymphatic tissue. This suggests the appendix may play an immunolgicaal role, weakening the claim that completely vestigial. Step 4: Choice D relates appendix size to reproductive fitness. This does not directly address whether the appendix is vestigial.
Step 5: Choice C provides the strongest evidence against the appendix being completely vestigial, by suggesting it has an immuunllogccl function.
Solution: Choice C
Solution Correctness: incorrect
Model Solution First. Error Step: 2
Model Solution Error Reason: Step 2 incorrectly assumes that the fact that the appendix might. have been larger in fossil hominids weakens the claim that the appendix is vestigial organ. If the appendix was larger in fossil hominids, it might indicate that it had some function in the past, but now these functions have diminished or become unimportant, which supports the claim that the appendix is vestigial organ. This option only indicates the function of the appendix in fossil hominids and cannot strongly prove the function of the appendix in modern humans.
Model Solution Rectified First Error Step: Step Option B indicates that the appendix might have been larger fossil hominids. This suggests that the appendix might have had more important functions evolutionary ancestors, further proving the claim that it vestigial organ. Step 2: Option indicates that the appendix might have been larger fossil hominids. However, this docs not explain the fanction of the appendix modern humans, so this option has limitations. Model Evaluation Result
Evaluated Model: gpl-4-turbo
Evaluation Raw Response:
Solution Analysis: - Step 1: This step is correct. It accurately identifies that the ability to remove the appendix without immediate ill effects supports the idea that the appendix might be vestigial. Step 2: This step is also correct. It suggests that if the appendix was larger in the past, it may have had a more significant function in fossil hominids, which provides a basis for it not being entirely vestigial in humans today.
Step 3: This step is correct. It presents the argument that the appendix's immunological function suggests it may serve purpose and is not completely vestigial.
Step 4: This step is neutral. It accurately states that Option D does not directly address whether the appendix is vestigial, which docsn't detract from nor support the idea of it being vestigial. Step 5: This step correct concluding that considering the provided options, the immunological role indicated by C (substantial amount of lymphstic tissue) provides the strongest evidence against the appendix being completely vestigial.
Solution Correctness: correct
First Error Step: N/A
Error Reason: N/A
Need Error Reason Review: False
Figure E.3-1. A failure case of the biology subject. It is another occurrence where the sampled LLM (Claude2) reached a correct answer through incorrect multi-step reasoning. The evaluator GPT-4 did not spot the knowledge and logic reasoning error in step about Choice B, but made a similar incorrect analysis.
25

---

Subject: college.biology
Question: Natural selection is based on all of the following except Options: [A.] genetic variation exists
within populations. [B.] the best-adapted individuals tend leave the most offspring, [C. individuals
who survive longer tend to leave more offspring than those who die young. [D] individuals adapt to thele enviromments and, therehy, evolve. Ground Truth Asswer: Sampled CoT
Sampled Model: GPT3.5
Model Solution Steps:
Stop I: Genetin variation within populations is a fundamental roncept of natural selection as it provides the raw material for evolution occur.
Step 2: The best-adapted individuals leaving the most offspring is a key aspect of natural selection as it leads to the passing on ad advantageuus traits to subsequent generations.
Step 3: Individusla who survive longer tend to leave more offepring than those who die young is directly related to the concept of natural selection, as individuals with advantageous traits are more likely to survive and reproduce.
Step 4: Individuals adapting to their environments and evalving is the core principle of natural selection, as organiams that are better suited to their environment are more likely survive and reproduce. Solution: Choice A: genetic variation exists within populatious.
Solution Correctness: incorrect
Model Solution First Error Step: 1
Model Solution Error Reason: The model that describes individuals adapting to their euvironments and evolving is incorrect are the basic units of evolution. The result of natural selection leads to species adapting to environments and evolving, but it Model Solution Rectified First Error Step: Step 4: Individuals adapting to their environments and evolving is not a core principle of natural selection, because individuals adapting to environments to increase their chances of survival and reproduction cannot be termed evolution. Solution: Option D: Individuals adapt to their enviromments and thereby evolve. Step 4: Individuals adapting to their envirouments and evolving is result of natural selection, Solution: Option D. Model Evaluation Result
core principle of natural selection, because it populations, not individuals, that
based the factor that individuals adapt to their enviroaments and thereby evolve.
Evaluated Model: gpt-4-turbo
Evaluation Raw Response:
Solution Analysis: Step 1: This step is correct as it accurately identifles that genetic variation within populations is essential for natural selection because it provides the necessary diversity which selection can act. Step 2: This step is correct and highlights that the best-adapted individuals generally reproduce more, which is fundamental to netural selection because ensures the spread of beneficial traits. Step 3: This step is correct, establishing that individuals who live longer have more opportanities to reproduce, contributing to the mechanistn of natural selection through differential survival. Step This step incorrect. It misstates the process of natural selection by suggesting that individuala adapt to their environments and thereby evolve. Natural selection acts on existing variations within a population and doesn't I involve organisms individually adapting evolving in response Solution Correctness: incorrect. First Error Step: Step Error Reason: Step 4 incorrectly implies that individual organisms adapt their environmenta and evolve as result, which misunderstands natural selection. In reality, natural selection operates on a population level based on existing genetic variatious, and individuals do not adapt or evolve during their lifetimes. Evolutionary changes can occur the population over generations as those with beneficial traits have better survival and reproductive success. The corrected reasoning for this step would be: "The misconception that individuals adapt and thereby evolve is not a principle of natural selection, natural selection acts heritable traits that beneficial prior and independently from environmental pressures, not post-adaptation by individuals."
environment.
Solution Correctness: incorrect
First Error Step: Step 4
Error Reason: Step 1 incorrectly implies that individual organisms adapt to their environments and evolve result, which misunderstands natural selection. In reality, natural selection operates level based on existing genetic variations, and individuals do not adapt or evolve during their lifotimess Evolutionary changes occur in the population
population
generations as those with beneficial traits have hetter survival and reproductive success. The corrected reasoning for this step would be: "The misconoeption that individuals adapt
thereby evolve not principle of natural selection, natural selection acts on heritable traits that are beneficial prior to and indepeudently from environmental pressures, not post- adaptation by individuala."
Need Error Reason Review: True
Error Reason Correctness Analysis: 1. Step-hy-Step Reasoning: - The original incorrect step implics that individual organisms are capable of adapting and evolving within their to environmental pressures. This is fundamental misunderstanding of how natural selection works. Natural selection operates at the level of populations over generations, not individuals. It the differential survival and reproduction of individuals with certain heritable traits that leads to evolution. The correct understanding is that while individuals with certain traits may have better survival and reproduction rates, these traits must already exist within the genetic makeup of the population. The population evolves over time as these traits beenme more common, not individuals adapting or evolving on their own.
lifetimes response
2. Student Error Reason Analysis: The student identifies the core issue with the incorrect step, noting that it incorrectly attributes the process of adaptation and evolution to individual organisms rather than populations. The student correctly points out that natural selection stion within population and that individuals themselves do not adapt or evolve during their lifetimes. The student's 's explanation correctly aligns with the principle that evolutionary changes are a consequence of differential reproductive surcess among individuals with different heritable traits, and these changes accumu- late over generations within a population. - The student accurately clarifies the misconoeption by explaining that natural selection acts heritable traits beneficial prior any environmental changes and that the process does not involve adaptation evolution by individuals.
based on pre-existing genetic vari-
3. Final Decision: Correct
Figure E.3-2. A success case of the biology subject.
26

---

E.4 Medicine
Question UUID: 4990050--20004445574334477721200
Subject: college medicine
Question:
A 63-ycar-old woman is in the hospital recovery room 4 hours after elective left hemicolectomy for colon carcinoma at the splenic flexure. She has preoperative written directive for no blood products through transfusion for religious reasons. Medical history is significant for hypertension and coronary artery disease. The nurse tells you the patient's ' bload pressure has progressively declined since the operation. Vital sigus now are temperature 35.8"C (96.4'F), pulse 130 130/min, respirations 20/min, and blood pressure 80/50 Hg. Physical examination discloses slightly distended abdomen with intact incision. ECG shows sinus tachyeardia. Urine output has been 10 mL during the past 2 hours. Hematocrit. is 30%; preoperative hematocrit was 41%. The patient has received I, of intravenous crystalloid in the recovery room. Reported operative blood loss 200 mL. Drainage from the naso- gastric tube is clear. Damage which of the following structures most likely responsible for these findings?
Options:
A: Epigastric artery
B: Liver
C: Middle colic artery
D: Spleen
Ground Truth Answer: D
Ground Truth Answer Modified: C
Sampled CoT
Sampled Model: mistral-medium
Model Solution Steps:
Step 1: Assess the vital signs and hematocrit
The patient's vital signs show hypotension (low blood pressure), tachycardia (rapid heart rate), and hypothermia (low body temperature). The urine output is also low, indicating possible renal hypop- erfusion. The significant drop in hematocrit (from 41% to 30%) suggests acute blood loss anemia. Step 2: Evaluate the intraoperative and postopcrative fluid management
The reported opcrative blood loss only 200 mL, which is not enough cause such significant drop in hematocrit and hemodynamic instability. The patient. received L. of intravenous crystalloids in the recovery room. However, this amount fluid should have compensated for blood loss and maintained blood pressure stability unless there is ongoing or occult blood loss.
Step 3: Assess the abdominal examination for signs internal bleoding The slightly distended abdomen and clear drainage from the nasogastric tube do not suggest an acute mechanical obstruction or ileus. The intact incision and absence of overt. bleeding also do not indicate an external source of hemorrhage.
Step 4: Consider possible injury to abdominal structures during the operation Since the patient had a left hemicolectomy, the potential structures that might have been injured include the epigastric artery, middle colic artery, spleen, or liver.
Step 5: Analyze each choice to determine the most likely cause of the patient's 's condition Choice_A: Epigastric artery- Injury to the epigastric artery is unlikely to cause significant bleeding during a left hemicolectomy, as these vessels not in the immediate operative field.
Choice B: Liver Although liver injury can lead to hemorrhage, it would typically present with acute onset of hypotension and peritoncal signs. The patient's 's initial vital signs stable, and there no evidence of peritonitis.
Choico. C: Middle colic artery This artery is in the operative field and could be injured during left hemicolectomy. Major bleeding from this artery could lead the patient's hemodynamic instability and hematocrit. drop.
Choice_D: Spleen Splenic injury could lead significant hemorrhage, but this is au uncommon com- plication of a left hemicolectomy due to the spleen's anatomic location.
findings and determine the most likely cause The patient's ' hemodynamic instability, low hematocrit, and minimal reported blood loss suggest on- going or occult. internal bleeding. The intact incision, clear nasogastric tube drainage, and slightly distended abdomen do not support external bleeding or mechanical obstruction. Among the possible injured structures, the middle colic artery is the most likely cause of the patient's 's condition. Solution: Choice_Co Middle colic artery
Step 6: Integrate
Solution Correctness: incorrect
Model Solution First Error Step: 2
Model Solution Error Reason:
The analysis mentions that the patient has received 4 liters of intravenous crystalloid in the recovery room, yet the blood pressure continues decline, indicating ongoing occult blood loss. The de- in hematocrit, which might be due to factors other than blood loss such hemodilution, has not been fully considered.
When the patient woke up from the surgery and was transferred the recovery room, the doctor had
27

---

already administered liters of crystalloid solution intravenously, which is equivalent to significant hydration. However, despite this hydration, the patient's blood pressure has not returned to normal and continues to drop. This could mean that there is ongoing or stealthy blood loss inside the patient's 's body, causing the blood pressure to decrease. Model Solution Rectified First Error Step:
Step 2: The patient has received 4 liters of intravemous crystalloid in the recovery room, but the blood pressure continues to decline. The reported operative blood loss was 200 mL, and the hematocrit dropped from 41% to 30%, which may be partly due the hemodilution effect of intravenous injection. Considering the patient's hemodynamic status, the possibility of ongoing or occult internal bloeding cannot be ruled out.
Step 2: After being transferred to the recovery room, the medical team administered 4 liters crystal- loid solution intravenously, which is equivalent to a significant amount of hydration. However, despite this, the patient's ' blood pressure continues to decline. The reported blood loss during the surgery was 200 mL, which usually would not cause a drop in blood pressure. Also, the patient's hematocrit dropped from 41% to 30%, which may be partly because the infused fluid diluted the blood. Model Evaluation Result.
Evaluated Model: gpt-4-turbo
Evaluation Raw Response:
Solution Analysis:
Step 1: Assess the vital sigus and hematocrit Correct. This step accurately summarizes vital signs and hematocrit values suggesting hypotension, tachycardia, hypothermia, and potential acute blood loss.
Step 2: Evaluate the intraoperative and postoperative fluid management- Correct. Observations correct in stating the discrepancy between reported blood loss and observed symptoms, raising ques- tions about additional or occult bleeding.
Step 3: Assess the abdominal examination for signs internal blooding Correct. The abdominal examination findings are evaluated logically to rule out external bleeding sources and acute mechanical ohstructions.
Step 4: Consider possible injury to abdominal structures during the operation Correct. Considering potential injuries to relevant abdominal structures based on the type of surgery performed is a logical step.
Step 5: Analyze each choice to determine the most likely cause Correct. Each option is analyzed based on anatomical relevance and likelihoad of contributing to the observed clinical picture, leading to rational elimination of less likely causes. Step 6: Integrate all findings and determine the most likely cause Correct. This step synthesizes the information from previous steps and identifies the middle colic artery as the most likely source of on- going internal bleeding, which is consistent with the location of the surgery and the clinical symptoms. Solution Correctness: correct
First Error Step: N/A
Error Reason: N/A
Solution Correctness: correct
First Error Step: N/A
Error Reason: N/A
Need Error Reason Review: False
Figure E.4-1. faihure case in the medicine subject. The question is relatively lengthy, highlighting the potential importance of multi-step reasoning for decomposition and analysis. Before diving into the comparison of four options, the sample model solution first provided the initial four steps to assess important information in the question context (e.g., patient status and treatments). The model solution contains error in step 2 analyzing the patient's s fluid management which the evaluator GPT-4 fails to identify.
28

---

Subject: college.tmedicnee
Question:
Which of the following diagnostic techniques is of no value for the diagnosis of neural tube defecte?
Options:
A: Amniocentosis
B: Chorion villus sampling (CVS)
C: Maternal serum screening
D: Uttasenography
Ground Truth Answer: B
Ground Truth Anawer Modified: Sampled CoT
Sampled Model: GPT3.5
Model Solution Steps:
Step 1: Amaiocentesis involves analyzing the amniotic fluid surrounding the developing fetus for genetic abnormalities. It detert neural tube defects and commonly used for prenatal diagnosis. Step 2: Chorion villus sampling (CVS) is another prenatal diagnostic technique that involves taking a sample of the placental tiasue to analyze the genetic composition of the fetus. It also detect neural tube defecta. Step 3: Maternal serum screening is a blood test that measures certain substances in the mother's blood to assess the risk of certain birth defects, including neural tube defects. It a valuable tool for screening but may not definitively diagnose neural tube defecta.
Step 4: Ultrasooggrapyy is a prenatal imaging technique that can visualize the developing fetus in the uterus.
It dotrett neural tube deferts by visualixing the spine and hrain of the frtus.
Solution: Choice_O- Maternal serum screening is of no value for the diagnosis of neural tube defects.
Solution Correctness: incorrect
Model Solution First Error Step: 2
Model Solution Error Reason:
The model incorrectly assumes that Chorionic Villus Sampling (CVS) can detect neural tube defects, CVS is primarily used for the detection of chromosomal abnormalities, not for diagnosing structural birth defects like neural tube defects.
Chorionic Villus Sampling is medical procedure that involves analyzing fetal cells in placental tiasue to detect chromosomal abnormalities and genetic diseases. Although this technique provide important infor- mation about the genetic condition of the fetus, its primary application does not include the direct diagnosia of neural tube defects. Therefore, describing the main purpose of CVS as directly used for diagnosing neural tube defects is inaccurate.
Model Solution Rectified First Error Step:
Step 2: Chorionie Villus Sampling (CVS) is prenatal diagnostic technique primarily used for detecting chromosomal abnormalities, such as Down syndrome, by analyzing placeutal tissue. It is not typically used to detect neural tube defects.
Step 2: Chorionic Villus Sampling (CVS) involves collecting placental tissue and analyzing fetal cella, and is not directly used for diagnosing neural tube defects.
Model Evaluntion Result-
Evaluated Model: gpt-4-turbo
Evaluation Raw Response:
Solution Analysis:
Step 1: Correct This explanation accurately describes the process and utility of amniocentesis in detecting genetic abnormalities, including neural tube defects.
Step 2: Incorrect Chorion villus sampling (CVS) primarily looks at chromosomal information and is more effertive in diagnosing chramosomal abnormalities rather than neural tube defictt, which best deterter using other methods like amniocentesis, maternal serum screening, or ultrasonography. Step 3: Correct- The description of maternal serum screening accurate. While the test valuable for screening, it is not definitive for diaguosis, hence its utility in assessing risk rather than contirming neural tube defects.
Step 4: Correct- Ultrasonography is correctly described visualization method capable of detecting ral tube defects through imaging.
Solution: Incorrect- The conclusion that maternal screening is of no value for the diagnosis of neural tube defects is incorret. Though it is not definitive for diagnosis, it is still a valuable screening tool and does provide relevant information suggest presence neural tube defects. Solution Correctness: incorrect
First Error Scep: Step 2
Error Reason: The error in Step pertains to the oversimplified or wrong explanation regarding the capa- bility of CVS to detect. neural tube defects. CVS is rather specific for detorting genetic and chromosomal abnormalities rather than structural defects like neural tube issues which are better detected by amniocente- sis or ultrasonography, A rectified response could state that while CVS is valuable for chromosomal analysis, does not typically detect neural tube defects, which better assessed by other diagnostic methods out- lined here.
Solution Correctness: incorrect First Error Step: Step 2
Error Reason: The error in Step 2 pertains to the oversimplified or wrong explanation regarding the ca- pability of CVS detect neural tube defects. CVS is rather specific for detecting genetic and chromosomal abnormalities rather than structural defects like neural tube issues which are better detected by amniocente- sis ultrasonograhhy. A rectified response could state that while CVS valuable for chromosomnal analysis, it does not typically detect neural tube defects, which are better assessed by other diagnostic methods out- lined here.
Need Error Reason Review: True Error Reason Correctness: Correct
Figure E.4-2. A success case of the medicine subject. The evaluator GPT-4 correctly identified the first error step and provided appropriate and rational reasons.
29

---

E.5 Chemistry
Subject: college.chemistry
Question: Calculate the magnetic moment (jy) of 18C mucleus.
Options:
A: 6.1445 x 10-27 T
B: 3.1445 x 10-27 J TT
C: 9.1445 x 10-27 J J
D: 1.1445 x 10-28 T
Ground Truth Answer: C
Ground Truth Answer Modified: A
Sampled CoT
Sampled Model: Claude2
Model Solution Steps:
Step I: The magnetic moment (ny) of nucleus dependa its spin quantum number (I) according to the following formula:
#1 7/h
Where the gyromagnetic ratio and à the reduced Planck constant.
Step 2: For 19C mucleus, I= -
Step 3: The gyromagnetic ratio for a 19C nucleus is 6.728 x 10" T-1a11
Step 4: Substituting the values for / and into the formula:
μı (6.728 10" 1())
Step 5: The value the reduced Planck constant 1.05457 x 10-34 calculation:
Step 6: Performing
#r = (6.728 10" 1 (==)g=))))(L0555555
10 34 J.s)
= 3.1144 x 10 J T=
Step 7: Comparing to the answer choices, the magnetic moment calculated matches Choice B.
Solution: Choice
Solution Correctness: incorrect
Model Solution First Error Step:
Model Solution Error Reason:
The formula given the steps incorrect, the correct formula should be jay -IA, where L (I)++))
The formula provided the steps is incorrect, the correct expression should be pr = yLh, where L=1(I+1).
Model Solution Rectified First Error Step:
The magnetic moment (ạr) of the nuclous is determined by the following formula based on its spin quantum number (I): μı = LL,, where L (I(1 + 1))²/2,
Step 1:
Step 1: The nuclear magnetic moment (μr) can be calculated using the following formula based on its spin quantum number (I): μ. = LL, where L represents L = I(I 1).
Model Evaluation Result-
Evaluated Model: gpt-4tuurbo22224-000099
Evaluation Raw Response:
Solution Analysis:
Step 1: This step is correct. It states the relationship between the muclear magnetic moment (17)), the gyromagnetic ratio (7), and the reduced Planck constant (f).
Step 2: This step is correct. The muclear spin quantum number (f) of carbon-13 (19C) indeed 3.
Step 3: This step is correct. The stated gyromagnetic ratio for a 13C nucleus (6.728 x 107 T-1s" is the correct value for this isotope.
Step 4: This step is correct. The given formula μj -T is being correctly used to substitute in the values of the gyromagnetic ratio (7) and the spin quantum number (7).
Step 5: This step is correct. The provided value of the reduced Planck constant (h 1.05457 10
J-s) is accurate.
Step 6: This step is correct. The calculation of the magnetic moment (μI) correctly performed using the provided values and the formula from step
Step This step is correct. The calculated result from step accurately compared with the options given in the question and matches choice B.
Solution Correctness:
First Error Step: N/A
Error Reason: N/A
Solution Correctness: correct
First Error Step: N/A
Error Reason: N/A
Need Error Reason Review: False
Figure E.5-1. A failure case in the chemistry subject.
30

---

Subject: high.scchoo..chemistty
Question:
Three 25.00 mL samples of approximately 0.10 M phenol, OHH,OH, K2 1.0 10-10 were removed from a container and placed separate mL beakers. The samples were titrated with standard potassium hydroxide, KOH, solution. Cresol red was the acid-base indicator used in the titration. The samples required 31.75, 32.38, and 41.75 mL to reach the endpoint. Which of the following might explain why one of the samples required significantly more base to reach the endpoint? Options:
A: The indicator was added too late.
B: The wrong indicator was used.
C: There was a base contaminating the unclean beaker.
D: There was an acid contaminating the unclean beaker.
Ground Truth Answer: D
Ground Truth Answer Modified: D
Sampled CoT
Sampled Model: Claude2
Model Solution Steps:
Step The phenol solutions were approximately 0.10 M concentration. At this concentration, the volumes KOH required reach the endpoint should be similar for all 3 samples.
samples required similar volumes KOH (31.75 mL and 32.38 mL). However, the third sample required significantly more (41.75 mL).
Step 2: Two
Step 3: Choice A suggests the indicator was added too late. This would not change the volume of KOH required, only the observed color change.
Step 4: Choice suggests the wrong indicator was used. This may affect the observed color change, but not the volume of KOH required.
Step & Choice C suggests a contaminating base in one sample. A contaminating base would react with the phenol, requiring more KOH to reach the endpoint.
Step 6: Choice D suggests contaminating acid. An acid contaminant. would be neutralized by the KOH and not affect the volume required.
Step 7: Therefore, the explanstion that contaminsting base required more KOH to reach the end- point in one sample is choice C.
Solution: Choice_C C
Solution Correctness: incorrect
Model Solution First Error Step:
Model Solution Error Reason:
The statement the erroneous step that KOH needed to reach the endpoint is inaccurate, The reaction of phenol with a basic contaminant would lead decrease amount of KOH used, not an increase.
In the description the crroncous step, the statement that more KOH is neoded to reach the endpoint is inaccurate. fact, the reaction of phenol with a basic contaminant would result: in a decrease in the consumption of KOH, not an increase.
Model Solution Rectified First Error Step:
Step 5: Option C indicates that there is a contaminated base in one of the samples. The contaminated base reacts with phenol, reducing the amount of KOH needed to reach the endpoint.
Step 5: Option C points out that the sample contains a contaminated base. This contaminated base reacts with phenol, leading to a reduction in the amount of KOH needed to reach the endpoint.
Model Evaluation Result
Evaluated Model: spt-4-turbo22024-01--55
Evaluation Raw Response:
Solution Analysis:
Step is correct. establishes the expectation for the volumes of KOH required based the concentration of phenol.
Step 2 correct. paints out the observed discrepancy in the volumes of KOH required, which prompts the question.
Step correct. Adding indicator too late would not alter the required volume of KOH, only the point at which the change observed.
Step is correct. Using the wrong indicator may affect the endpoint's observation but not the actual chemically required volume KOH.
Stee is incorrect. The explanation here misunderstands the effect of contaminating base. A con- taminating base in sample would actually reduce the amount of KOH required to reach the endpoint because it would partially neutralize the acid (phenol) present before the KOH is added, not require more KOH as stated.
Step is also incorrect but the identified first error is in Step 5. A contaminating acid would indeed require more KOH to reach the endpoint because it adds to the total amount of acid that must be neutralized.
Step 7 reaches incorrect conclusion based incorrect reasoning presented in Step 5. Solution Correctuess:: incorrect First Error Step: Step Error Reason: The error in reasoning is that contam-
31

---

inating base would lead to increased consumption of KOH. In reality, contaminating base would have already neutralized some of the acid present, thus requiring less KOH to reach the endpoint of the titration. The corrected reasoning should suggest that contaminating acid (not base) in the unclean beaker would explain why KOH required reach the endpoint, pointing toward Choice D as the correct explanation rather than Choice C.
Solution Correctness: incorrect
First Error Step: Step
Error Reason: The error reasoning that a contaminating base would lead an increased consumption KOH. In reality, contaminating base would have already neutralized some of the acid present, thus requiring less KOH to reach the endpoint
titration. The corrected reasoning should suggest that a contaminating acid (not hase) in the unclean beaker would explain why more KOH was required to reach the endpoint, pointing toward Choice D as the correct explanation rather than Choice C.
Need Error Reason Review: True
Error Reason Correctness Analysis:
Scoring Model: gpt-4-turba-20244-04--8
Scoring Raw Response:
1. Step-by-Step Reasoning:
'The error identifies the misconception about the effect of a contaminating base in of the samples. is incorrectly stated that contaminating base would lead increased consumption of KOH to reach the cudpoint.
In reality, if base contaminant had reacted with the phenol, it would have neutralized some of the acid, thereby reducing the amount of KOH necessary reach the endpoint.
The correct understanding that a contaminating acid, a base, would require additional KOH to neutralize, thus explaining increased volume of KOH needed for one the samples.
2. Student Error Reason Analysis:
The student correctly identifies the error in reasoning regarding the effect a contaminating base on the consumption KOH.
They accurately point out that a contaminating hase would result less, not more, KOH needed to reach the titration endpoint because the base would have neutralized some of the acid present.
The student correctly redirects the reasoning towards a contaminating acid being the cause for the increased KOH volume required, aligning with the true reason for the observed discrepancy.
This explanation correctly shifts from Choice C (a contaminating base) to Choice D (a contaminating acid) as the valid reason for the anomaly in the titration results.
3. Final Decision: Correct
Annotation Analysis:
The error identifies the misconception about the effect of contaminating base in of the samples. incorrectly stated that contaminating basc would lead an increased cousumption of KOH to reach the cudpoint.
In reality, if base contaminant, had reacted with the phenol, it would have neutralized some of the acid, therehy reducing the amount of KOH necessary reach the endpoint.
The correct understanding that a contaminating acid, not a base, would require additional KOH to neutralize, thus explaining increased volume of KOH needed for one the samples.
Error Reason Analysis:
The student correctly identifies the error in reasoning regarding the effect a contaminating base on the consumption KOH.
They accurately point out that a contaminating base would result in less, not more, KOH needed to reach the titration endpoint because the base would have neutralized some of the acid present.
The student correctly redirects the reasoning towards a contaminating acid being the cause for the increased KOH volume required, aligning with the truc reason for the observed discrepancy.
This explanation correctly shifts from Choice C' (a contamiuating basc) to Choice D (a contaminating acid) as the valid reason for the anomaly in the titration results.
Error Reason Correctness: Correct
Figure E.5-2. A sucoess case in the chemistry subject.
32

---

E.6 Logic
Question UUID: 57a04b12-h958- 405
Subject: logic
Question:
A highway management office decided to recruit 10 auxiliary road managers to solve the problem of shortage of formal managers, but this proposal aroused opposition from some people. The person believes that with the addition of these 10 road auxiliary managers, there will be more road violations and disciplinary violations will be found, and more formal managers be needed deal with these problems the later stage. This will loadl to a shortage of already tense formal managers. Which of the following, if true, will most weaken that person's is point of view?
Options:
A: The newly recruited auxiliary road managers may not be able to do their job with due diligence
B: There are many road violations, and problem disciplinary violations be solved the spot,
without delaying to post-processing
C: Auxiliary road managers can also deal with road violations and disciplinary violations later
D: The increase of road auxiliary managers will effectively reduce the occurrence of road violations and disciplinary violations on this section of the road
Ground Truth Answer:
Ground Truth Answer Modified: C
Sampled CoT
Sampled Model: GPT3.5
Model Solution Stops:
Step I: If the newly recruited auxiliary road managers may not be able to do their job with due diligence, could potentially lead to more road violations and disciplinary issues, supporting the opposition's 's point of view.
Step 2: If there are many road violations, and the problem of disciplinary violations can be solved on the spot without delay, it weakens argument that more formal managers will be needed to handle the issues later.
Stop 3: auxiliary road managers can also deal with road violations and disciplinary violations later, it weakens the argument that more formal managers will be required to handle these problema.
Step 4: If the increase of road auxiliary managers will effectively reduce the occurrence of road violations and disciplinary violations, it completely weakens the opposition's point of view that more formal managers will be needed to handle the expected increase in violations. Solution: Choico_D
Solution Correctness: incorrect
Model Solution First Error Step: 4
Model Solution Error Reason:
Option D does not completely weaken the opponent's point of view. Because it only says that incrensing personnel can effectively reduce violstions, but amount of reduction uncertain, and the impact on later work is also uncertain. Therefore, the final solution reasoning result is incorrect. This does not completely weaken the opponent's 's point view. Because it
specific number of how increasing personnel can effectively reduce violations, it does not provide the most reliable evidence. Model Solution Rectifled First Error Step:
Step 4: If the added auxiliary road managers disciplinary violations, this to some extent weakens the opponent's 's point of view. But because it only says that increasing personnel can effectively reduce violations, the amount of reduction is uncertain, and the impact on later work is also uncertain. Therefore, in comparison, option C is the most weakening option. Solution: Option C.
effectively reduce the occurrence of road violations and
Step 4: If the added auxiliary road managers can effectively reduce the occurrence of road violations and disciplinary violations, this weakens the opponent's 's point of view, that is, more formal managers are needed to deal with the expected increase violations. But it lacks the specific number how increasing personnel can effectively reduce violations, it does not provide the most reliable evidence. Option C states that even if violations are discovered, auxiliary personnel can also solve the problem, so it will not increase workload of formal personnel. Therefore, it most weakens the point of view. Solution: Option C. Model Evaluation Result-
Evaluated Model: gpe-4-turbo-2221-1--09
Evaluation Raw Respouse:
Solution Analysis:
Step 1 explains Option A in which the suxiliary road managers may not perform well, potentially increasing issues rather than reducing them. This step neutral as it provides background Option A but does not directly weaken the opposition's is viewpoint.
Step 2 discusses Option B, suggesting that immediate resolution violations reduces the need for formal managers later. This ressoning correctly identifles how this option weakens the argument. This step is correct.
Step examines Option C, indicating that suxillary road managers being able handle violations later could also weaken the need for formal managers. This is a valid point, and this step is correct.
Step analysis Option D and concludes that directly weakens the oppasition's point by reducing occurrence of violations. This analysis is correct with sound reasoning.
Thee selected solution (Choice D) directly addreases the concern about incressing violations requiring more formal managers by positing that violations would decrease. Solution Correctness: correct First Error Step: N/A Error Reason: N/A
Solution Correctness: correct
First Error Step: N/A
Error Reason: N/A
Need Error Reason Review: False
Figure E.6-1. A failure case in the logic subject.
33

---

Question UUID: 177477--66ib--311----------B
Subject: logic
Question:
Living gifts refer to the donation of money to children, grandchildren and other relatives when the donor decides to live, or use for vacations and the estahlishment of trust funds. Among the following options, which is living gift?
Options:
A: Mr. Wang made will before his death, leuving 90% of his property to his only granddaughter.
B: Chairman Li handed over 80% of his property and the entire group to his daughter, and then lived in the mountain village by himself.
C: Grandma Wang asked her grandson to get married in her own house and live in nursing home by herself.
D: Two years after Mr. Li disappeared, his son inherited his property. One day, he suddenly appeared at home.
Ground Truth Answer:
Ground Truth Answer Modified: B
Sampled CoT
Sampled Model: GPT3.5
Model Solution Steps:
Step 1: Mr. Wang leaving 90% of his property to his only granddaughter in his will is not a living gift as it occurs after his death.
Step 2: Chairman Li handing over 80% his property to his daughter and then living in a mountain village is living gift as it also
before his death.
Step 3: Grandma Wang asking her grandson to get married in her own house and living in nursing home by herself not living gift, but more of familial arrangement.
Step 4: Two years after Mr. Li disappeared, his son inherited his property, which suggests it was not planned living gift.
Conclusion: The only option that fits the definition of living gift is Choice.A: Mr. Wang made a will before his death, leaving 90% of his property to his only granddanghter.
Solution Correctness: incorrect
Model Solution First Error Step: 2
Model Solution Error Reason:
The model incorrectly assumes that Chairman Li handing over 80% of his property to his daughter is not living gift. According to the defiuition of living gifts, these gifts are given when the donor is alive, and option B describes Chairman Li giving his property to his daughter during his lifetime, which fits the definition of a living gift. Therefore, the judgment. in step 2 is wrong.
Model Solution Rectified First Error Step:
Step 2: It should be pointed out that option B describes a living gift because Chairman Li transferred his property his daughter during his lifetimee which Step 2: Option B describes case of living gifts because Chairman Li transferred his property his daughter during his lifetime, which is consistent with the definition of living gifts, that is, the donation is made when the donor alive.
with the definition of living gifts.
Model Evaluntion Result-
Evaluated Model: gpt-A-turbo Evaluation Raw Response:
Solution Analysis:
Step 1: This step correctly identifies that Mr. Wang's will, which comes into effect after his death, does not qualify as a living gift. The reasoning is sound because living gifts are defined as douations made while the donor is still alive.
Step 2: This step misinterprets the definition of a living gift, Chairman Li handing over 80% of his property to his daughter and then living elaewhere matches the definition of living gift since is done while he is alive. Thus, the conclusion drawn in this step is incorrect.
Step 3: The analysis here is correct in determining that Grandma Wang's action doesn't 't directly involve gift to live; instead, it's about living arrangement. However, it could
transfer of money or property still be considered a form of living gift indirectly, but the explanation does not explore this possibility. The neutral due to lack of exploration.
step is
Step 4: The identification that Mr. Li's post-disappeared inheritance does not constitute a living gift is correct, as relies on the absence and presumed death of Mr.
Conclusion: The couclusion drawn is incorrect, as it identifies Option A as the correct answer, which was established in the first step as not fitting the definition of living gift due to taking effect after death. Solution Correctness: incorrect
First Error Step: Step 2
Error Reason: The first error occurs Step where the solution incorrectly states that Chairman Li handing over property to his daughter is not a living gift because it "occurs before his death." This contradicts the very deftuition of living gift, which is specified to the donation money, property, etc., to relatives while the donor is still alive. Chairman Li's action of transferring his property to his daughter while he is alive perfectly aligus with the concept of a living gift. The corrected reasoning for Step 2 would affirm that gift, directly reflecting the definition provided in the question.
Chairman Li's action is indeed
Solution Correctness: incorrect
First Error Step: Step 2
Error Reason: The first error occurs in Step 2. where the solution incorrectly states that Chairman Li handing over property to daughter is living gift because "oceurs before his death." This coutradicts the very defluition of living gift, which is specified to be the donation of money, property, etc., to relatives while the douor is still alive. Chairman Li's action of transferring his property to his daughter while he is alive perfectly aligns with the concept of living gift. The corrected reasoning for Step 2 would affirm that Chairman Li's action is indeed a living gift, directly reflecting the definition provided in the question. Need Error Resson Review: True Error Reason Correctness: Correct
Figure E.6-2. A success case in the logic subject.
34

---

E.7 Coding
Subject: coding
Question:
Write Python function according the function nsme and the problem description in the docstring below. fron typing ixport List
uwximu.eizzeaaiie
(ouns1: List [int], mune2: [int(:
нv* You are given two nemory quantities numsl and nuns2 whose subscripte start from D, and their lengthe both even
You must delete / 2 elamenta from numal and / alementa fron muna2. After deletion, you insert the renaining elementa from nural and nuna2 into the set Raturns the maximum nunbar of possible collactions ##*
Bos.inuSSSeeeefoeeiavaa (1,2])
aac.muusssaaaaffrrooooo......... [1,1,1,11)
Sampled CoT
Sampled Model: GPT3.5 Model Solution Steps:
maximun atza_aftar..r...oaaoa/aaaaaaaaae ttt_aaaatrrrorraalllleaaa renoval (numal: [int] nuna2: Liat[int]]:: Sort the lista and the maximun collection nunsi. ()
length Len(nunst))
Calculate the number of elemente to delete which 18 / remove_count n //
Create a set from the concatenated lists removing elements as required (renove_count:)
(renove_count:)]
Return the naximun number elementa the reaulting collection aet return (en(s)
cases
print (naximun_tiae sectiimn__iieeeeeeeee siwe_aftere___eeeeeeeeeeeeeeeeee aeteeerrenovaa(S 4), [1, 2))) # print (maximun setee_aftr__roeee after_ronoval(I. 2, 1, 2], [1, 1, 1)))
Solution Correctness:
incorrect
Model Solution First Error Step:
# Return the maximum number of elements in the resulting collection aet
Model Solution Error Reason:
The provided code has problem because it does not appropriately select which elements to remove from 'numsl' and 'nums2" first n/2 elements from each array, which does not necessarily lead to the maximum possible unique set size. Especially, the smaller elements are freguently repeated and just trimmed away, this method could result in maximizing the size of the final
size
unique
remaining elements.
simply
Model Solution Rectifedd First Error Step: # Find the combination of elements remove that maximizes the size of the resulting
Evaluation
Evaluated Model: gpt-4-turbo
Evalustion Raw Response:
Solution Analysis:
1. The code starts by sorting the 'numsl' and 'nums2' lists. Sorting is neutral in this context since ensuring inherently cause an error.
not necessary but
2. It then determines the length 'n' of 'numsl" I' (assumed to he equal to 'nums2' since both are said to he of even length 'n"). This is correct logic because length is necessary to compute number elements to be removed.
The code calculates the mumber elements to deleted, 'remove count' as // 2. Given that 'n' is always
question), this is 4. It then creates set 's' by concatenating 'numsl' and 'nums2' from indices vemove_count onwards. This thus working on the latter half of
stipulated
computation ensuring integers for indexing.
clements from 5. Finally, it returns the size of sot 's' which represents the number of unique elements in the resulting collected elementa. This operation correctly used by 'len(s)'. The provided implementation perfectly resonates with the objective of the algorithm explained by the docstring. Solution Correctness: correct
First Error Step: N/A
Error Reason: N/A Need Error Reason
Figure E.7-1. A failure case in coding. The sampled code fails to identify the key point that the set is unindexed, hence duplicate elements will be removed. The asscssed GPT-4 fails to annotate the when evaluating the code solution, only blindly accepting the code logically correct and computable.
35

---

Quextion: Write Python function according to the function name and the problem description in the docstring below.
from typing inport List def perange_ark.paite **.
deck: List [int]) int:
Legend tells of great Ark built by Noah survive immense flood world, invited pairs so
would cover Earth. preserve
join on Ark, inviting species could contiaue in the new sorld. the animals bearded they were assigned places in linear formation across the Ark's deck. However, in the rush to board before the rain began, the animal paizs became separated across the 2n available spaces. Each aninal is known by unique identifier, and the 1ist of these identifiere as they are arranged on the Ark is given by an integer array 'arkDeck" where "arkDeck [1]' zepresente the animal occupying the ith apace. The paizs were meant to board in order, with the first pair being (0, 1), the second pair (2, 3), and so
Ark,
to the last pair being (2n - 2, 2n 1)
Your task is to help Noah figure out the minimus number of exchanges necessary to reposition the animals that each pair is resting side side. An exchange is the of two aninals, regardless of their species, standing up from their places and switching spots the deck. >>> (rrange.rrk_pairrr00....333
>>> arrange_ (
Ground Truth Anewer: Sampled Model: CPT3.6
Model Solution Steps:
from typing import List
def arrange_ark.paiie dark)_deck: [ist[int]) svaps Initialize
анaрs
needed
n len (arkaa // Calculate the number of pairs based on the deck size
# Iterate over the pairs and check if they are in order or not for 1 in range(u):
if drkk_ [2+1] == 2+1 and ark)_ (2+1+1) == 2+1+1: continue # If the pair is in order, move to the next pair else:
If the pair is not in order, find the position of the second member of the in the deck
for in range (2+1+1, (on(ark\_
if ark ark)_deck[]] 2+i+1:
Swap the positions of members of the pair [2-11]1) arkk_deck[j] ark)_deck[j]] ark) deck [2+1+1]
swaps break
Increment the swap count by
return внaps
Test cases
print perrrang_ark_paaiee ((0,1,3,2))
Output: 0 (No awaps needed as pairs are in order)
print (arrange_ark.paitg 2,,
Output: 1 (1 correct the pair positions)
needed
Model Solution Correctnessi Incorrect
Model Solution Error Step: continue & pair order, move next pait
Model Solution Reason: attompta verify whether udjuceat pains meet roquirements. state-
position
required
3,3) pair, (p.J.0.1]
pair.
36

---

Model Solution Rectifed First Error Step:
arck_deck
1] arc)_ (2
1] + 1 or ark\_deck [2 i]
ark [ea\_ecck[ (2 1 1] 1:
Model Evaluation Result
Evaluated Model: gpt-i-turbo
Evaluation Result:
Solution Analysia:
1. The fination signature and the Import statement are correct and ses the right context for the problem
2. Initialiaing 'swaps" to 'O' is correct as it preparts to count the number of awaps neccussry this consrbbutigg to solving the problem.
3. Calenhating se "lan dack) // 2 correctly determines the number of paire, given the problem's is constrainte.
4. The 'for' loap starting with 'for in range(n) sete up iteration over mac pair correctly
conditional arkadeck221 == 2t: and ark == 24+4: incorrect. The logic asrumes that the pairs being the
order solely dependent their Indices being even and odd sequentially, which doce not align paine
reganding problem block starting iteration of
idantifted aranged
"2"111 look
pair
spproach find swap misglaced
The awapping logic inside the 'clse' block incrementing the 'swapo"
by awap made logically sound.
contributing achieving the task
Returning 'awaps' correctly provider the solution to the problem based on the accumulated value.
9. The provided test cases confirm the solution's approuch under specific conditions but might not aufficiently teat all odge canee.
Solution Correctnece: incorrect
First Eeror Stop: Harkkacckk' ii] - 2 CondurRudeekip + + -- 2+
solution correctnoss: incorreot
error reason: This line incorrecsly anumes that the correct order pairs determined by checking indices and values match such that both members the pair indiees and values even and add sequentially (Lee. and 21 The describer
problem
pair
being reguential
paire (e-g. neceonrtlly (2, ste.), position, siguifionatly modifying
posttions eequentiall
comparing correction involves The solution should Instead locate the partner of oquation derived from its own ideatifier, comidering the identifeers determine the paizz. This involves correcting the logie to fiud and swap the unimal correctly, which might remove or alter this conditional check cutirely favor of au approuch this considars the valuss of the identiflers rather than their positions in the array.
idensifier appreach animals. checks pains based their values and their positions in deck:
considering the
condition
animal caleslating
expected partner's identifier weing
Need Error Reason Review:
True
Error Resson Scoring
Evaluated Model: gpt-4sturbo
Evsluxtion Raw Respanse:
Step-by-Ster Rewsenin::
The geound trath otror that conditional cleck the code insorrectly assumes auimals must aequeatial order by their identifiers
eto.). deek (i.e. (0, 1) must positions and (2. 3) at positions positions 12 and 22"+41"
The provided
decke
exactly '2t)' and "2"i+1'.
the dack. This problem that as long specifics
only need be pair's idensiflers adjaomtt the order acceptable.
side does mandate specific order
the condition iff ark deck(p)))" == 2 and dekceec(2)) == 2444. This condition falle to check the adjacent numbers are paits regardleso of their order on the deck.
The
The rectified step suggests alsernnsive appronch that veritles it adjaceut elements are pars of the same pait by checking their values, nos their posltions.
2. Student Error Reason Analyoia:
Thhe student correctly identifler the firzt error step in the solucion as the candition that checks if the indices and valuee match in specific sequential pastern.
Thee student explains that the solution should check pairz based on thair values rather than their positions, This aligned with she ground trush the paits only need adjucent, not in a specific sequential position on dook. adjacons numbers being pairs identifiers
or modify condision ensure student's proposed correction This implica change that then positiona the array.
value
Final Decision: Corract
Annotation Analyala:
Thee ground trath utror is that the conditional check in the code incorrectly assumes that animals muat be aequential order by their identiflers on the deck (La., (0, 1) must be at positions o and 1, (2, 3) at pocitions 2 and 3, eto.).
TThe provided solation sterster over the elemants of wark decke in pains and checks if the valees at positions '2tp and '2"141" are exucily '2"3' and (2% 13'.
However, the problem statement specifics that pains ouly need to be side by side and does not mandaie a opecific order ou the dock, This meons that as long an cock pair's 's idensifters are adjacen,, the order accepiable.
The error in the condition ark d)ck(2*1 == 24 and ark [ == 24543.. This condition fails to check adjacet numbers are paits regurdleas of their order on the deck. step suggests alsernative approsch that verifles adjacet preitione.
are part same pair by chocking
values,
Reason Analysia
The studens correctly identiffes the first step the solution match specific sequential pactern.
condition that checks the indices and values
The student explains that the solution should check for pairs based on their valuea rather than their positions. This is aligned with the ground trusk thas the paits only need to be adjuceat, The student'' x propored correction to ramove or modify the condition to ansure it checks for adjacans nurbhers baing pairs based on their idantifire is also corract. This implies a logic change that focusee on the valuon of the animal identifiers rather than their positions in the array.
in specific sequential position on the dock.
Error Reason Correctneee: Correct
Figure F.7-2. A success case in the coding subject. The model evaluates the error part correctly, and its error reason annotation aligns with human annotations.
37

---

G Computational Resources Used
In this paper, all experiments either performed on open-source models with local inference or closed-source models with API calls. For local inference, we are using A800 machines with 8 GPUs to run the inferences. The total evaluation time on our 6k benchmark on the 70B language models typically takes around 2 hours using fast inference libraries such as vllm. For smaller language models such as Phi-3 or Gemma, the compute time is smaller.
H Annotation Guidelines
Below provide the original annotation guidelines distributed to annotators of distinctive subjects included the MR-Ben benchmark: math biology phyyiccs] chemistry| logic medicine and codding The guidelines serve as the primary trainigg material and instructions for annotators to complete the labeling tasks, specified with detailed descriptions, requirements, and standards.
43

---

Annotation Guideline: Math Subject
1 Background
With the increasingly enhanced capabilities large language models such as ChatGPT, existing models can break down complex problems and solve them step by step. However, while their solutions may often be seemingly correct, they contain logic. In this annotation task, annotators will evaluate moodll-generaterr step-by-step solutions questions in the mathematics subject (including university- and high-school-level mathematics), and determine whether the model has obtained the correct answer based on the correct understanding and reasoning logic.
in understanding, calculation, and reasoning
2 Annotation Task
Given a question and a stp-py-step solution generated by the model, the annotator needs to read the question and models for question, and they need to be annotated independently. Each annotation task is divided into four sub-tasks.
solutions carefully. There will be different solutions from multiple
2.1 Sub-task 1
The first sub-task is to decide whether the goodel-generated step-by-step solution is correct. Even the model's final answer matches the ground-truth answer, the model may arrive at the correct answer through incorrect understanding and reasoning. Therefore, regardless of whether the final answers match the ground-trude choice, annotators should read through each model solution carefully and complete all subtasks based on prudent judgments.
2.2 Sub-task 2
The second sub-task is to identify the first incorrect step in the moner-generatte solution if the solution is incorrect. Specifically, each step can be interpreted as correct, neutral, or incorrect.
Correct means both the calculation and understanding correct, and the step is essential for reaching correct result.
Neutral refers the current step possibly being background statement about the question, a factual/formula description, or an exploratory step generated by the model. There are no obvious errors in this step, but whether this step leads to the final correct answer is not obvious.
Incorrect step refers to obvious errors in calculation, understanding, other possible errors reasoning, regardless of whether this error affects the final calculation result.
Please note that we only need to find the first 'incorrect' step.
2.3 Sub-task 3
The third sub-task is to describe the reason for the error in the first incorrect step. Annotators should carefully analyze the possible reasons for the error in the incorrect step and provide rational analysis. Please try to incorporate the logic of the model's solution and analyze the reasons for this error from
39

---

a holistic perspective. This enrrr reason may serve as reference for other annotators or models the future, so please be as detailed, comprehensive, and clear as possible.
2.4 Sub-task 4
The fourth and final sub-lask is to rewrite the first incorrect step. Based on the emor reason you provided, correct the incorrect step that you identified. The rewritten step should naturally integrate into the original context of the model solution, with only the incorrect part modified when necessary.
3 Annotation Data Format
At the end of this guideline, Figure I shows an annolation example of one model-generrtedd solution for high-school mathematics question. which includes the following items:
Question, Options, Model Solution Steps and Ground Truth Answer: They are what annotators should carefully read and analyze.
Model Solution Correctness is the first sub-task item to be annotated. It is defined as whether the model solution is completely correct for all steps. Please only answer "Yes" or "No" in this column.
Please do fill in other variants such as correct, incorrect, positive, negative, etc.
Model SSouution First_ Error SeerrStte__uuue is the second sub-task item be annotated. It defined as the first incorrect step specified "No", please enter the numerical step numher with quotation marks here, such "I", "2", etc.; If the result of the first item annotation is "Yes", fill in "None" here. Please note:
model solution. the annotation of the previous item
1. Do not fill in other variants such as "one", "two", "none". "no". etc.
Step numbers should be written in double quotation marks; do not just write numbers.
Model Solution Error Reason is the third sub-task item be annotated. Please provide detailed error analysis for the identified first incorrect step here. For example, whether the model refers to incorrect factual knowledge/ correct facts but incorrect comprehension, calculation incorrect contextual logic links / improper conditions assumptions, etc. If the annotation of the first item "Model Solution Correctness" is "Yes", fill in "None" here.
Rectified Model Solution First Error Step is the fourth and last sub-task item to be annotated. Please correct the incorrect step based on the error reason you provided. If the annotation the first item "Model [oldellSSouutin... Correctness" is "Yes", fill in "None" here. Note that we not responsible for the consequential steps following the first incorrect step; we only need correct the identified first error here.
4 Annotation Regulations & Quality Control
All questions are to be annotated according to the format requirements. If issues arise with the annotation format from sampling inspection, annotators should make modifications promptly.
If the annotations of error reasons and step corrections are too brief, perfunctory, or unclear, re- annotation is required to ensure the annotation quality. For each batch submitted files, if any of the following occurred, all questions of the same batch will be sent back for re-ex re-examination and re-annolation:
1. the annotation error rate is higher than 5% after random sampling, any the requirements this guideline are found violated.
5 Additional Notes for Math Subject
5.1 Look-Up Source
The mathematics subject of this task includes annotating questions in the difficulty level of American university and high school mathematics. When encountering unfamiliar or unclear mathematical
40

---

terms concepts. please use search engines and conduct factual inquiries from authoritative sources to assist with your annotation. Please do not use any language models as your annotation assistance tool. Contact the annotation management team whenever in doubt.
5.2 Math Expressions Format
The writing of mathematical formulas and special symbols is in the LaTeX language.
1.
For convenience in question interpretation, unnotators
website tools such as https: //www.latens www.exiieiieee..mm com/ to convert and view content enclosed in the symbols the question. Since latexlive.com uses a single backslash instead of double backslash, please change the two backslashes (W) inside the mathematical formula in the question (e.g., formula.
(23) single backslash (1) (e.g., $Mfaac (23) when previewing the
Correspondingly. when filling in the annotations, annotators need to convert and write the mathematical formulas: please change the single backslash (1) back a double backslash (1) and cnclose the mathematical formula with. Examples of specific filling can be seen in the sample question annotation.
6 Common Questions
Q1: When a step repeats the incorrect option should I mark it as the first incorrect step?
No, if a step only describes what an incorrect option is stating or repeats the exact words of the incorrect option without analysis, does not constitute incorrect step. But if the step includes wrong analysis/judgment about the incorrect option, then it considered incorrect.
Q2: If previous steps so far have only stated relevant facts without specific analysis, how should I handle these steps?
Similarly, if the steps focus solely on factual statements about the options related background descriptions, these steps should be understood as neutral steps unless these steps contain incorrect understanding, incorrect facts, etc. The first incorrect step should be the one that involves incorrect analysis or reasoning.
Q3: What should I do if encounter difficulty comprehending the question options or model solutions?
When encountering unfamiliar terms, look up textbook sources search engines to find reliable explanations first. Contact the annotation management team whenever doubt.
Q4: The model's answer gives two answers, one of which is correct. How should decide?
Annotate it as "No." Then follow the standard process to annotate the incorrect step, error reason, and the correction step.
Q5: It's inconvenient to read the JSON file without line breaks.
If you are using platforms like Visual Studio Code to annotate files in JSON format, you can install plugins like 'json editor' to assist in reading. However, take caution against modifying any content other than the required subtasks and do not manually change the data structure.
7 Annotation Example
Figure below is model solution of sample question to be annotated; Figure 2 shows example annotation for the same model solution.
41

---

"guseston, 200.51
*Quesston to noives that fecarth) Niteset:C413,, (eidev2s) (7 Zore -5, riad fiaroral
Midrani5) locs (1frac126112) sabt (1 7y
Goeenn Teath Inacyate": sn, aod Trath "o Tol
seter Recros by
S1 trrm the other van vniing the ge ses opert oo
*Ppan (bo2e) (1E0ae/7) (915, ve get 20 (iseectiiiiieeriss
"Step subacitoce the ecpressioe for and foand in step deco the equasion for (eince it bas ste conuat tecma) eqatim "t ciidract7 le 20 2p 101 13 1116.
Sirplity equation
stap 2.", the eqatioo the Geooalsataes",
I fcantoos
11)
sibeaciztell3 (1 th15
(itreci2t00 13 Jax25) 15.5,
noe do casa, fless reareange the equatics to the standaed quadeatic foom ih,
Wiree) optic
anter
saecaca that Witeact
siepaitioation, obtaies"
Vireaci 12111 (2)sessgtii Mizaaci (0 (2a)) 03.", aclutiins.
costinue
eteer Susseitore the expressiors for and o fourd in Stops 1 and b reapective.y yy into the epection for b. froe Step 1. then
"heeall that do Wireac (01122015 Sabatstute the expresatins for and free Deape and inte tate squation, then solve
"etey Misally sduats ate the colations Cor and fosed in steps by Josi tropestiinly into the expretaiion
iivalcace abanlzbe vatee
categ found anlutionn tcom powrioze ataps."
<Teep compaze the peesit
vasa the given Chosos chotze and Cholze to diss ite ratching option
cheics lost choice
foom peeclous
"Seep' Modelf, "elecral meddants
"Mods: Secesion, Erecs Desashh; "E2D",
ssactiiles Rode: socesion, Fozat, Secoe Secoe_ Stap": "E2D"
Figure 1: task example of model-geneeattd solution for high-school mathematics question (to be annotated).
"Model Solatioon Correctness's "so",
"Nodel Sol lon First Reroor_tteg_eeeee Bin": +3",
"Model solattionn breus Beasonn; *Step has al slinination and shitimiic eesoz when wlininating Cosntions
by mileipl ying the denoninator: ielnsting fract lone by mule holleplyingg the denoninator 8-2 2 Nirrraili 04:-01244-33333 (310332222 111
yields -7 etc 2/30strcc(1C4 (s 261 812,, not 8-7 200 (24 HD) ote 20ffree (4 330133 1133.".
213trac/1041 (6428)))
2. Fizst, eliminate the Fraation ion by sultiglying the denominators 8-5 dto a 2 (911))))) 1919, Repanding and lyppiifynng furthee,
"Rectiried Modell_Goouttoe First Stop") "Simpiity the set equation
8- ofd (20ae (3 (41b))
Figure 2: An example of standard annotation (with the previous items omitted save space).
42

---

Annotation Guideline: Physics Subject
1 Background
With the increasingly enhanced capabilities large language models such as ChatGPT, existing models can break down complex problems and solve them step by step. However, while their solutions may often be seemingly correct, they contain errors in understanding, calculation, and reasoning logic. In this anmotation task, annotators will evaluate modell-geneattee step-by-step solutions to questions in the physics subject (including university-lever and high-school-level physics), and determine whether the model has obtained the correct answer based on the correct understanding and reasoning logic.
2 Annotation Task
Given a question and a stp-py-step solution generated by the model, the annotator needs to read the question and models for question, and they need to be annotated independently. Each annotation task is divided into four sub-tasks.
solutions carefully. There will be different solutions from multiple
2.1 Sub-task 1
The first sub-task is to decide whether the goodel-generated step-by-step solution is correct. Even the model's final answer matches the ground-truth answer, the model may arrive at the correct answer through incorrect understanding and reasoning. Therefore, regardless of whether the final answers match the ground-trude choice, annotators should read through each model solution carefully and complete all subtasks based on prudent judgments.
2.2 Sub-task 2
The second sub-task is to identify the first incorrect step in the moner-generatted solution if the solution is incorrect. Specifically, each step can be interpreted as correct, neutral, or incorrect.
Correct means both the calculation and understanding correct, and the step is essential for reaching correct result.
Neutral refers the current step possibly being background statement about the question, a factual/formula description, or an exploratory step generated by the model. There are no obvious errors in this step, but whether this step leads to the final correct answer is not obvious.
Incorrect step refers to obvious errors in calculation, understanding, other possible errors reasoning, regardless of whether this error affects the final calculation result.
Please note that we only need to find the first 'incorrect' step.
2.3 Sub-task 3
The third sub-task is to describe the reason for the error in the first incorrect step. Annotators should carefully analyze the possible reasons for the error in the incorrect step and provide rational analysis. Please try to incorporate the logic of the model's solution and analyze the reasons for this error from
43

---

a holistic perspective. This enrrr reason may serve as reference for other annotators models the future, so please be as detailed, comprehensive, and clear as possible.
2.4 Sub-task 4
The fourth and final sub-lask is to rewrite the first incorrect step. Based on the emor reason you provided, correct the incorrect step that you identified. The rewritten step should naturally integrate into the original context of the model solution, with only the incorrect part modified when necessary.
3 Annotation Data Format
At the end of this guideline, Figure I shows an annolation example of one modell-gnneaated solution for a physics question, which includes the following items:
Question, Options, Model Solution Steps and Ground Truth Answer: They are what annotators should carefully read and analyze.
Model Solution Correctness is the first sub-task item to be annotated. It is defined as whether the model solution is completely correct for all steps. Please only answer "Yes" or "No" in this column.
Please do fill in other variants such as correct, incorrect, positive, negative, etc.
Model SSouution First_ Error SeerrStte__uuue is the second sub-task item be annotated. It defined as the first incorrect step specified "No", please enter the numerical slep numher with quotation marks here, such "I", "2", etc.; If the result of the first item annotation is "Yes", fill in "None" here. Please note:
model solution. the annotation of the previous item
1. Do not fill in other variants such as "one", "two", "none". "no". etc.
Step numbers should be written in double quotation marks; do not just write numbers.
Model Solution Error Reason is the third sub-task item be annotated. Please provide detailed error analysis for the identified first incorrect step here. For example, whether the model refers to incorrect factual knowledge correct facts but incorrect comprehension, calculation incorrect contextual logic links / improper conditions assumptions, etc. If the annotation of the first item "Model Solution Correctness" is "Yes", fill in "None" here.
Rectified Model Solution First Error Step is the fourth and last sub-task item to be annotated. Please correct the incorrect step based on the error reason you provided. If the annotation the first item "Model [oldellSSouutin.. Correctness" is "Yes", fill in "None" here. Note that we not responsible for the consequential steps following the first incorrect step; we only need correct the identified first error here.
4 Annotation Regulations & Quality Control
All questions are to be annotated according to the format requirements. If issues arise with the annotation format from sampling inspection, annotators should make modifications promptly.
If the annotations of error reasons and step corrections are too brief, perfunctory, unclear, re- annotation is required to ensure the annotation quality. For each batch submitted files, if any of the following occurred, all questions of the same batch will be sent back for re-ex re-examination and re-annolation:
1. the annotation error rate is higher than 5% after random sampling, any the requirements this guideline are found violated.
5 Additional Notes for Physics Subjects
5.1 Look-Up Source
The physics subject of this task includes annotating questions in the difficulty level of American university and high school physies. When encountering unfamiliar unclear terms and concepts,
44

---

please use search engines conduct factual inquiries from authoritative sources to assist with your annotation. Please do not use any language models as your annotation assistance tool. Contact the annotation management team whenever doubt.
5.2 Physics Expressions Format
The writing of physics formulas and special symbols is in the LaTeXX language.
1.
For convenience in question interpretation, annotators can website tools such https://wwe http///wwwwat.eeeeeeee// com/ convert and view the content enclosed in the sym- bols in the question. Since latexlive.com uses a single backslash instead of double buckslash, please change the two buckslashes (1) inside the physics formula in the question (e.g., (23 ( (3]S) single backslash (1) (c.g., SM (23) (3}S) when previewing the formula.
Correspondingly, when filling in the annotations, annotators need to convert and write the physics formulas: please change single backslash (1) back double backslash (1) and enclose the physics formula with. Examples of specific filling can be seen in the sample question annotation.
6 Common Questions
Q1: When a step repeats the incorrect option should I mark it as the first incorrect step?
No, if a step only describes what an incorrect option is stating or repeats the exact words of the incorrect option without analysis, does not constitute incorrect step. But if the step includes wrong analysis/judgment about the incorrect option, then it considered incorrect.
Q2: If previous steps so far have only stated relevant facts without specific analysis, how should I handle these steps?
Similarly, if the steps focus solely on factual statements about the options related background descriptions, these steps should be understood as neutral steps unless these steps contain incorrect understanding, incorrect facts, etc. The first incorrect step should be the one that involves incorrect analysis or reasoning.
Q3: What should I do if encounter difficulty comprehending the question options or model solutions?
When encountering unfamiliar terms, look up textbook sources search engines to find reliable explanations first. Contact the annotation management team whenever doubt.
Q4: The model's answer gives two answers, one of which is correct. How should decide?
Annotate it as "No." Then follow the standard process to annotate the incorrect step, error reason, and the correction step.
Q5: It's inconvenient to read the JSON file without line breaks.
If you are using platforms like Visual Studio Code to annotate files in JSON format, you can install plugins like "json editor' to assist in reading. However, take caution against modifying any content other than the required subtasks and do not manually change the data structure.
7 Annotation Example
Figure below is model solution of sample question to be annotated; Figure 2 shows example annotation for the same model solution.
45

---

"qusstien, ouzer; "dblattel- bet.
"Bunject"; "hish schocl I physsest, "guestien"t body woving the posteive rerecctoon pasres the crigtn at cite O. Between t and second, the body has constane speed ot retere per secend, At recond, the body in givenn constant acoslaration mntare per secend equared in the nagative direction, the posittion x tae body at reconda dat,
"optiosatt "Aa 99s 36x Aads +adx. And: +8im " knalyete"t
"gcound Souch Anseat Modified"; *C*)
"Baxpled model"; "nlasxol-rediant,
solucion Stepa";
interval disection. dispiacurant **,
ada) bedy
apead
"Slay 2: application
accund, the velocity acceleration of o/s? in zegazive direction.",
boly mosemiarrly fly
doe the instantaroouu
"Step 3: For the rubrequnnt Line pariod Eren
aucend to 11 auconda, the body'a nction ia gevarned by vas bast with C,
conatent scouleration the nequtive dizection. Daing the kineratic equasinn 0, -6 niat, and L 10 auconta (since 11 - 10), find -300 x.",,
"seep io neveves, the dieplacerant calcuisted on Step 3 seecs to on acded to the claplacemett tound in Step 1, az both displacarants are in the same diraction. This, the Fsnal positiio a in (26 m) 1-100 of -276 a.",
re,
"Supp St mons of the candidata choiess satch, the calculated valwe in Stop 6. Since the unsoee aust be and of the provided cloccaa, theze seure to be a zissaka us useos in the quntioo as the avstlable cheiceo, We cannot dateanine the correct solution bazed on the given infomation,",
*Botations Unable to detemnine colucion ueiin the given chofces."
1.
mMdde] Soturton Dorrectrees resor
"Mudul solucion Fisst, Easos Blup": "sec",
"Mudel solution Beaot Reason"s "ssu", "Xodel solucion feotifled Flest Beroz Stopfs
Figure I: A task example of model-generated solution for a high-school-level physics question (to be annotated).
"Nodel Solation, Correstness" "Sncorreat", Soluttoo "Modal" Solution Brvor Reasoa": "Roserding to the definition. acosicration: acsciorction guintity that desusibes how quickly the velocisy of an abjecs changes, Sombired with the probler statermet thas (htha object is eiven a cocstast accelesation of é natera par second aquazed in the negative x di direction,)* i it la know that this acceleration only casses the velucity to change graduelly and does rot nake the velauity
Step": "2*,
physical
bring the abject
stap"; instant stop.Ant
"Madel polution jeetified Fizat,
acculesation of neqative disection, ebjact's veloeity
aucond, due
applied Instantaeeuully first decesases gradially alone the posstiva x dizection until it reduces to 2025, then incrcases the negative x direation. An!
neters
Figure 2: An example of standard annotation (with the previous items omitted save space).
46

---

Annotation Guideline:
Biology Subject
1 Background
With the increasingly enhanced capabilities large language models such as ChatGPT, existing models can break down complex problems and solve them step by step. However, while their solutions may often be seemingly correct, they contain logic. In this annotation task, annotators will evaluate model-generatee step-by-step solutions to questions in the biology subject (including university-leve and high-school-Ieve biology), and determine whether the model has obtained the correct answer based on the correct understanding and reasoning logic.
in understanding, calculation, and reasoning
2 Annotation Task
Given a question and a stepp solution generated by the model, the annotator needs to read the question and models for question, and they need to be annotated independently. Each annotation task is divided into four sub-tasks.
solutions carefully. There will be different solutions from multiple
2.1 Sub-task 1
The first sub-task is to decide whether the goodel-generated step-by-step solution is correct. Even the model's final answer matches the ground-truth answer, the model may arrive at the correct answer through incorrect understanding and reasoning. Therefore, regardless of whether the final answers match the ground-trude choice, annotators should read through each model solution carefully and complete all subtasks based on prudent judgments.
2.2 Sub-task 2
The second sub-task is to identify the first incorrect step in the moner-generated solution if the solution is incorrect. Specifically, each step can be interpreted as correct, neutral, or incorrect.
Correct means both the calculation and understanding correct, and the step is essential for reaching correct result.
Neutral refers the current step possibly being background statement about the question, a factual/formula description, or an exploratory step generated by the model. There are no obvious errors in this step, but whether this step leads to the final correct answer is not obvious.
Incorrect step refers to obvious errors in calculation, understanding, other possible errors reasoning, regardless of whether this error affects the final calculation result.
Please note that we only need to find the first 'incorrect' step.
2.3 Sub-task 3
The third sub-task is to describe the reason for the error in the first incorrect step. Annotators should carefully analyze the possible reasons for the error in the incorrect step and provide rational analysis. Please try to incorporate the logic of the model's solution and analyze the reasons for this error from
47

---

a holistic perspective. This enrrr reason may serve as reference for other annotators or models the future, so please be as detailed, comprehensive, and clear as possible.
2.4 Sub-task 4
The fourth and final sub-lask is to rewrite the first incorrect step. Based on the emor reason you provided, correct the incorrect step that you identified. The rewritten step should naturally integrate into the original context of the model solution, with only the incorrect part modified when necessary.
3 Annotation Data Format
At the end of this guideline, Figure I shows an annolation example of one modellgennerated solution for a high-school biology question, which includes the following items:
Question, Options, Model Solution Steps and Ground Truth Answer: They are what annotators should carefully read and analyze.
Model Solution Correctness is the first sub-task item to be annotated. It is defined as whether the model solution is completely correct for all steps. Please only answer "Yes" or "No" in this column.
Please do fill in other variants such as correct, incorrect, positive, negative, etc.
Model SSouution First_ Error SeerrStte__uuue is the second sub-task item be annotated. It defined as the first incorrect step specified "No", please enter the numerical step numher with quotation marks here, such "I", "2", etc.; If the result of the first item annotation is "Yes", fill in "None" here. Please note:
model solution. the annotation of the previous item
1. Do not fill in other variants such as "one", "two", "none". "no". etc.
Step numbers should be written in double quotation marks; do not just write numbers.
Model Solution Error Reason is the third sub-task item be annotated. Please provide detailed error analysis for the identified first incorrect step here. For example, whether the model refers to incorrect factual knowledge/ correct facts but incorrect comprehension, calculation incorrect contextual logic links / improper conditions assumptions, etc. If the annotation of the first item "Model Solution Correctness" is "Yes", fill in "None" here.
Rectified Model Solution First Error Step is the fourth and last sub-task item to be annotated. Please correct the incorrect step based on the error reason you provided. If the annotation the first item "Model_ [oldellSSouutin.. Correctness" is "Yes", fill in "None" here. Note that we not responsible for the consequential steps following the first incorrect step; we only need correct the identified first error here.
4 Annotation Regulations & Quality Control
All questions are to be annotated according to the format requirements. If issues arise with the annotation format from sampling inspection, annotators should make modifications promptly.
If the annotations of error reasons and step corrections are too brief, perfunctory, or unclear, re- annotation is required to ensure the annotation quality. For each batch submitted files, if any of the following occurred, all questions of the same batch will be sent back for re-ex re-examination and re-annolation:
1. the annotation error rate is higher than 5% after random sampling, any the requirements this guideline are found violated.
5 Additional Notes for Biology Subject
5.1 Look-Up Source
The biology subject of this task includes annotating questions in the difficulty level of American university and high school biology. When encountering unfamiliar unclear terms and concepts,
48

---

please use search engines conduct factual inquiries from authoritative sources to assist with your annotation. Please do not use any language models as your annotation assistance tool. Contact the annotation management team whenever doubt.
6
Common Questions
Q1: When step repeats the incorrect option should I mark it the first incorrect step?
No. if step only describes what incorrect option is stating repeats the exact words of the incorrect option without analysis. does not constitute an incorrect step. But if the step includes wrong analysis/jddgment about the incorrect option, then it considered incorrect.
Q2: If previous steps far have only stated relevant facts without specific analysis, how should handle these steps?
Similarly, the steps focus solely on factual statements about options or related background descriptions, these steps should be understood as neutral steps unless these steps contain incorrect understanding, incorrect facts, elc. The first incorrect step should be the that involves incorrect analysis or reasoning.
Q3: What should I do if encounter difficulty comprehending the question options or model solutions?
When encountering unfamiliar terms, look up textbook sources or use search engines to find reliable explanations first. Contact the annotation management team whenever in doubt.
Q4: The model's answer gives two answers, of which is correct. How should decide?
Annotate it "No." Then follow the standard process to annotate the incorrect step, reason, and the correction step.
Q5: It's inconvenient to read the JSON file without line breaks.
If you are using platforms like Visual Studio Code to annotate files in JSON format, you can install plugins like 'json editor' assist in reading. However, take caution against modifying any content other than the required subtasks and do not manually change the data structure.
7 Annotation Example
Figure below is model solution of sample question annotated; Figure 2 shows an example annotation for the same model solution.

---

192725 1e--tel-40---oooo---
"dobjenthy the tohoel bie logy", "vusttinns "Reth
plants eaploy paysieioçica: and biochanioal atzaseqiea ceduce pistoreppiration. Wiset the
*A2 C4 2 sasappoaphaaee lag CS pianta fix co2 in the assepoyi: but caszy zut the caivin cycia in the bussia sheath, while cae plance dix pos and gaeey dee
0o2 the 4-carboo anid orsiorrttats, while GRl plante fix 002 3-caction acid zibvicse
sysia
pessptyck. satt planta dis in the purdle abeath sut caszy nut the dazeun cycls the phooptogy yoilate,
while oar plants store 000 2n osa csolearettee. vcroudd muth poa yolos
*Mode" Silut
'Stap 11 C4 planss fix 0O2 as the deston acid osaccacerabe, Moise CAN plance dix DOG as the cacton soud cibslcoe 3,
blaphoaphaca."
resophyli catry des the Dalvin oycle in the nepephyli by
"Sbep 24
the Calvis
buntle sheath, shill CAM
000 and
spacist csparation
opola plante
Snaccuratsly dosorihos tiis
plaats.
"step ti of plante fix cal ie the Sandle theath but carry out the osivis oycte ie the nesophyll, shile oow planta fix pud and catzy oet the Daivin oycle in the binele steath."
plante.", "Stap bn Thia atatemant traressabely descrioes the spaticl seperation initiel D02 timetion the Calvin cycte in
"step 71 of plants store 0C2 as phorphagyyoaatte while CAR planta store 202 so osstoacetate.""
"atep This atatemant inaccurataly dasoribes how 002 ss stoxed of and olt planse.",
"Diepled Mode." *g175.5",
Medes Intutier Breen Jhers 1980% ant Keeoe aeasact) "200",
Figure task example of model-generated solution for high-school biology question (to be annotated).
MMddel Solution torrectness"); "No",
model Solution First Froor Step Bus"; 1g*,
"todal Solution Eccos Reaoo": "The fixation of 0o2 to S-oachonate ribulese 1, L-bhhhhhhoppaat by cAt planto is an Incoczect statesency 002 is fixed in osalic acid by the onsyse BE carbosylase in Gil planta (he sare step used by 0t plants), and then it is converted to salfe aeid or sore other type of organie acid.", "Rectified Yodel solution First Froor Step") "The deseription of this stacesenc is incerrect, Richough d4 plants six 207 to oxslascerate
carbonate, photoresgi-attin "ation is vore livited in phorosyncheele to dak plance due to tha delayed kineties of enayee astivution, and 0o2 is fixed in malie acid rainly by DEe eactoxylase in cld planta ithe saze stop as that used by 04 plantol, and then is convected to ralie acid oz other kinda of organic acids,"
Figure 2: An example of standard annotation (with the previous items omitted to save space).
50

---

Annotation Guideline: Medicine Subject
1 Background
With the increasingly enhanced capabilities of large language models such ChatGPT, existing models can break down complex problems and solve them step by step. However, while their solutions may often be seemingly correct, they contain enrors in understanding, calculation, and reasoning logic. In this annotation task, annotators will evaluate model-gencrated step-by-step solutions questions the medicine subject, and determinc whether the model has obtained the correct answer based on the correct understanding and reasoning logic.
2 Annotation Task
Given a question and a step-by-step solution generated by the model, the annotator needs to read the question and model-gencrated solutions carefully. There will be different solutions from multiple models for one question, and they need to be annotated independently. Each annotation task is divided into four sub-tasks.
2.1 Sub-task 1
The first sub-task is to decide whether the mroel--ennerted step-by-step solution is correct. Even if the model's final answer matches the ground-truth answer, the model may arrive at the correct answer through incorrect understanding
reasoning. Therefore, regardless of whether the final answers match the ground- truth choice, annotators should read through each model solution carefully and complete all subtasks based on prudent judgments.
2.2
Sub-task 2
The second sub-task is identify the first incorrect step in the model gener-ggne solution if the solution is incorrect. Specifically, each step can be interpreted as correct, neutral, or incorrect.
Correct means both the calculation and understanding are correct, and the step essential for reaching the correct result.
Neutral refers to the current step possibly being a background statement about question, a factual/formula description, or an exploratory step generated by the model. There are obvious errors in this step, but whether this step leads the final correct answer is not obvious.
Incorrect step refers to obvious errors in calculation, understanding, other possible emror reasoning, regardless of whether this error affects the final calculation result.
Please note that only need to find the first 'incorrect' step.
2.3
Sub-task 3
The third sub-task is to describe the reason for the error in the first incorrect step. Annotators should carefully analyze the possible reasons for the error in the incorrect step and provide rational analysis. Please try to incorporate the logic of the model's solution and analyze the reasons for this error from a holistic perspective, This error reason may serve reference for other annotators models in the future, please be as detailed, comprehensive, and clear as possible.
51

---

2.4 Sub-task 4
The fourth and final sub-task rewrite the first incorrect step. Based on the error reason you provided, correct the incorrect step that you identified. The rewritten step should naturally integrate into the original context of the model solution, with only the incorrect part modified when necessary.
3 Annotation Data Format
the end of this guideline. Figure shows an annotation example of one model-generated solution for a medicine question, which includes the following items:
Question, Options. Model_Solution SteppssSttseste and Ground_ Ground_Truth Truthh_Aswwer They are what annotators should carefully read and analyze.
Model_ Saaiee__Sottto_____ Corrrcttess is the first sub-task item to be annotated. It defined whether the model solution is completely correct for all steps. Please only answer "Yes" or "No" in this column. Please do not fill other variants such as correct, incorrect, positive, negative, etc.
Model Solution First Error_Stee Step Num is the second sub-task item to be annotated. It is defined as the first incorrect step specified in the model solution. If the annotation of the previous item is "No", please enter the numerical step number with quotation marks here, such as "1", "2", etc.; If the result of the first item annotation "Yes", in "None" here, Please note:
Step_Num'>
Do fill in other variants such as "one", "two", "none", "no", etc.
2. Step numbers should be written in double quotation marks; do not just write numhers.
Model Solution Error Reason is the third sub-task item to be annotated. Please provide a detailed error analysis for the identified first incorrect step here. For example, whether the model refers incorrect factual knowledge/ correct facts but incorrect comprehension, calculation incorrect contextual logic links / improper conditions or assumptions, etc. the annotation of the first item "Model SalpelSSoouutooSeeeeeeeSCC Correettes"""""""""""" is "Yes", fill in "None" here.
Rectified Model Solution First Error Step is the fourth and last sub-task item to be annotated. Please correct the incorrect step based on the error reason you provided. If the annotation of the first item "Model_Solution Soluution_Corrccttessst Correctness" is "Yes", fill in "None" here. Note that we are not responsible for the consequential steps following the first incorrect step; we only need to correct the identified first error here.
4 Annotation Regulations & Quality Control
All questions are to anmotated according to the format requirements. If issues arise with the annotation format from sampling inspection, annotators should make modifications promptly.
If the annotations of error reasons and step corrections are too brief, perfunctory, or unclear, re- annotation is required to ensure the annotation quality. For each batch of submitted files, if any of the following occurred, all questions of the same batch will be sent back for re-cxamination and re-annotation:
1. the annotation error rate is higher than 5% after random sampling, or 2. any of the requirements in this guideline found violated.
5 Additional Notes for Medicine Subject
5.1 Look-Up Source
Annotating the medicine subject questions requires professional expertise the related field, as annotators have been notified and assessed accordingly in the recruitment round. When encountering unfamiliar or unclear terms and concepts, please use search engines and conduct factual inquiries from authoritative sources to assist with your annotation. Please do not use any language models as your annotation assistance tool. Contact the annotation management whenever in doubt.
52

---

6 Common Questions
Q1: When a step repeats the incorrect option should I mark it as the first incorrect step?
No, if step only describes what an incorrect option stating or repeats the exact words of the incorrect option without analysis, it does not constitule an incorrect step. But if the step includes wrong unalysis/judgment about the incorrect option, then it is considered incorrect.
Q2: If previous steps so far have only stated relevant facts without specific analysis, how should I handle these steps?
Similarly. if the steps focus solely on factual statements about the options or related background descriptions, these steps should be understood as neutral steps unless these steps contain incorrect understanding, incorrect facts, etc. The first incorrect step should be the one that involves incorrect analysis or reasoning.
Q3: What should I do if encounter difficulty comprehending the question options model solutions?
When encountering unfamiliar terms, look up textbook sources or use search engines to find reliable explamations first. Contact the annotation management team whenever in doubt.
Q4: The model's answer gives two answers, one of which is correct. How should decide?
Annolate as "No." Then follow the standard process to annotate the incorrect step, error reason, and the correction step.
Q5: It's inconvenient read the JSON file without line breaks.
If you are using platforms like Visual Studio Code annolate files in JSON format, you can install plugins like "json editor' to assist in reading. However, take caution against modifying any content other than the required subtasks and do not manually change the data structure.
7 Annotation Example
Figure below is model solution of sample question to be annotated; Figure 2 shows an example annotation for the same model solution.
*gusation, DUDD:; rSedtee2 7633 4372 4002 477460667314", toollape" "guest linet emoghhomaat ine resytthesle during rensery from evenel on te invibieed bye", exends evest on, loke hyperveee lon, becs on exceco of osggin Vots a Look osygen. "
"Goourd Truth Anavects 1p%,
MMode, Colecios Sbapat: 1
excreise, recyechests daring vceevery estop chotos a otates that as cxneee of orcatino unbibiie phorplocreatiee mncytteria. this is ooreect, as orcatire ecapates with phorptacacation for the coagee omarine Liasso shish needed for piospheeration serysthosin.', incurmet, iypersotstatiinn
stosphreeeat
recysthooise during
roqutcos
"stop
bypecontitation
scepechonis.
affect eaergy
ratabalion.
*Step Chosos atatea in escees of exypen inosbits cesynthasis, This La incorrect, as oxypen is requiced for secobie energy production dering zanovery.
sepyrotwrin,
oxygen
energy
cotsbalion to piosido ensroy for phorglaciesstn rosyttoriis
*Soep Chotoes and are correct. in sccess creatine and Lack oeypen beth intibit phosptocrestion resynsberia dezing recovazy *Soiction: Choice
"Seepues Modsif; *C.eide2",
"Modal Jolution Flest.
*Hodal solution krace Reaacer "Eab", Specti find godel ocsecion, Firas Recop Brepe_Stap"t Staph: 1000
1.
Figure 1: A task example of model-generated solution for medicine question (to be annotated).
53

---

"Model Solution Cocreotness"' "No", "Model Solution,
"Model Ssiution Rroor Reseon": "Excees creatine is srrociated with inhibstiion phorpherrettine casynthanis,",
"Rectiffedd Vodel wow.ttonn_Pteeeeeoooooo First, Error, Sreph:: 2: Option suggeste that creatine fahibite sesynchesio, This incocrecty ereatine suplexentation ineseases axount ezeatine pools ciasue and Increases the asount of phosphocreacine, shich ixproves the ability to zesynchestre during Incensity axazcias."
Figure 2: An example of standard annotation (with the previous items omitted save space).
54

---

Annotation Guideline: Chemistry Subject
1 Background
With the increasingly enhanced capabilities large language models such as ChatGPT, existing models can break down complex problems and solve them step by step. However, while their solutions may often be seemingly correct, they contain logic. In this annotation task, annotators will evaluate model-gencrater step-by-step solutions to chemistry), and
in understanding, calculation, and reasoning
questions in the chemistry subject (including university-level and determine whether the model has obtained the correct answer based on the correct understanding and reasoning logic.
2 Annotation Task
Given a question and a stppp solution generated by the model, the annotator needs to read the question and models for question, and they need to be annotated independently. Each annotation task is divided into four sub-tasks.
solutions carefully. There will be different solutions from multiple
2.1 Sub-task 1
The first sub-task is to decide whether the goodel-generated step-by-step solution is correct. Even the model's final answer matches the ground-truth answer, the model may arrive at the correct answer through incorrect understanding and reasoning. Therefore, regardless of whether the final answers match the ground-trude choice, annotators should read through each model solution carefully and complete all subtasks based on prudent judgments.
2.2 Sub-task 2
The second sub-task is to identify the first incorrect step in the moner-generated solution if the solution is incorrect. Specifically, each step can be interpreted as correct, neutral, or incorrect.
Correct means both the calculation and understanding correct, and the step is essential for reaching correct result.
Neutral refers the current step possibly being background statement about the question, a factual/formula description, or an exploratory step generated by the model. There are no obvious errors in this step, but whether this step leads to the final correct answer is not obvious.
Incorrect step refers to obvious errors in calculation, understanding, other possible errors reasoning, regardless of whether this error affects the final calculation result.
Please note that we only need to find the first 'incorrect' step.
2.3 Sub-task 3
The third sub-task is to describe the reason for the error in the first incorrect step. Annotators should carefully analyze the possible reasons for the error in the incorrect step and provide rational analysis. Please try to incorporate the logic of the model's solution and analyze the reasons for this error from
55

---

a holistic perspective. This enrrr reason may serve as reference for other annotators or models the future, so please be as detailed, comprehensive, and clear as possible.
2.4 Sub-task 4
The fourth and final sub-lask is to rewrite the first incorrect step. Based on the emor reason you provided, correct the incorrect step that you identified. The rewritten step should naturally integrate into the original context of the model solution, with only the incorrect part modified when necessary.
3 Annotation Data Format
At the end of this guideline, Figure I shows an annolation example of one modell-gnnerated solution for a high-school chemistry question, which includes the following items:
Question, Options, Model Solution Steps and Ground Truth Answer: They are what annotators should carefully read and analyze.
Model Solution Correctness is the first sub-task item to be annotated. It is defined as whether the model solution is completely correct for all steps. Please only answer "Yes" or "No" in this column.
Please do fill in other variants such as correct, incorrect, positive, negative, etc.
Model SSouution First_ Error SeerrStte__uuue is the second sub-task item be annotated. It defined as the first incorrect step specified "No", please enter the numerical step numher with quotation marks here, such "I", "2", etc.; If the result of the first item annotation is "Yes", fill in "None" here. Please note:
model solution. the annotation of the previous item
1. Do not fill in other variants such as "one", "two", "none". "no". etc.
Step numbers should be written in double quotation marks; do not just write numbers.
Model Solution Error Reason is the third sub-task item be annotated. Please provide detailed error analysis for the identified first incorrect step here. For example, whether the model refers to incorrect factual knowledge/ correct facts but incorrect comprehension, calculation incorrect contextual logic links / improper conditions assumptions, etc. If the annotation of the first item "Model Solution Correctness" is "Yes", fill in "None" here.
Rectified Model Solution First Error Step is the fourth and last sub-task item to be annotated. Please correct the incorrect step based on the error reason you provided. If the annotation the first item "Model_ [oldellSSouutin... Correctness" is "Yes", fill in "None" here. Note that we not responsible for the consequential steps following the first incorrect step; we only need correct the identified first error here.
4 Annotation Regulations & Quality Control
All questions are to be annotated according to the format requirements. If issues arise with the annotation format from sampling inspection, annotators should make modifications promptly.
If the annotations of error reasons and step corrections are too brief, perfunctory, or unclear, re- annotation is required to ensure the annotation quality. For each batch submitted files, if any of the following occurred, all questions of the same batch will be sent back for re-ex re-examination and re-annolation:
1. the annotation error rate is higher than 5% after random sampling, any the requirements this guideline are found violated.
5 Additional Notes for Chemistry Subject
5.1 Look-Up Source
The chemistry subject of this task includes unnotating questions in the difficulty level of American university and high school chemistry. When encountering unfamiliar unclear and concepts,
56

---

please use search engines conduct factual inquiries from authoritative sources to assist with your annotation. Please do not use any language models as your annotation assistance tool. Contact the annotation management team whenever doubt.
6
Common Questions
Q1: When step repeats the incorrect option should I mark it the first incorrect step?
No. if step only describes what incorrect option is stating repeats the exact words of the incorrect option without analysis. does not constitute an incorrect step. But if the step includes wrong analysis/jddgment about the incorrect option, then it considered incorrect.
Q2: If previous steps far have only stated relevant facts without specific analysis, how should handle these steps?
Similarly, the steps focus solely on factual statements about options or related background descriptions, these steps should be understood as neutral steps unless these steps contain incorrect understanding, incorrect facts, elc. The first incorrect step should be the that involves incorrect analysis or reasoning.
Q3: What should I do if encounter difficulty comprehending the question options or model solutions?
When encountering unfamiliar terms, look up textbook sources or use search engines to find reliable explanations first. Contact the annotation management team whenever in doubt.
Q4: The model's answer gives two answers, of which is correct. How should decide?
Annotate it "No." Then follow the standard process to annotate the incorrect step, reason, and the correction step.
Q5: It's inconvenient to read the JSON file without line breaks.
If you are using platforms like Visual Studio Code to annotate files in JSON format, you can install plugins like 'json editor' assist in reading. However, take caution against modifying any content other than the required subtasks and do not manually change the data structure.
7 Annotation Example
Figure below is model solution of sample question annotated; Figure 2 shows an example annotation for the same model solution.
57

---

*Subjeet*? Ssigh school "Questionty sanich of the tollowing to not good seadure of relative Interolecutar attractive forcest", racound couth Acalyais"; my
topesond*) SAT Rest of fostos youy softing points yodr Vapor presesree yaby Best reporiastion "y saroond Druth Inewesf; "Ma, "Mutel Solution Stapoft
stasngis interaclacciaa
ensrgy
change ribatance feem solid Ligli atste.",
"stop 21 soiling points are also good ressure of selative Intermolenslar attzactive forosa since subotances with acconoes cntarrolenciaz foccez zequire noza ensrqy to cvarcorm these dordsa and seii.", sgtep sapor presuures are another good weasure relative intermotecol attractive forces because sibstances tStep
regerisation higher vapor preesuves."
strength
energy required
"Stap daring all choices, be that all then meaccree relative tntervolacsiar attructive lorom. Eaazeoow, the shoivw Stnu."
"lapplad Kodal"; 10223.55,
solution Cocrentneso" m1007 Firet__eees___
meddll Solut Rvror Ressonts *Ten*, "Sectified Hedei Solution Firas Scpos Soupts "E00"
Figure 1: A task example of model generrted solution for high-school chemistry question (to be annotated).
"odde Solutioo Correctness"; "insorvec:", "oddel Solution, Pizst Ervor Brop"_ "1" three, "Hodel_Bolution
"Hadel, Solution, Ervor Recson": "The heat of susion escentially the same as the zemaining towards nothed compared
is liquid, naking not
transition solld other aptions.\a"
"Model Solution Rectifled Pirst Seror teror "Step it The beat of fuxion actually meaxates the enezgy auquired to diarupt the ordezly astangeae:: of solumcles in osyatal scructure, cather chan teat the atranath of intorsoleaciar attractive Yor"as.\a",
Figure 2: An example of standard annotation (with the previous items omitted save space).
58

---

Annotation Guideline: Logic Subject
1 Background
With the increasingly enhanced capabilities of large language models such ChatGPT, existing models can break down complex problems and solve them step by step. However, while their solutions may often be seemingly correct, they can contain errors in understanding, calculation, and reasoning logic. In this annotation task, annotators will evaluate model-gencratee step-by-step solutions to questions in the logic subject, and determinc whether the model has obtained the correct answer based on the correct understanding and reasoning logic.
2 Annotation Task
Given a question and a step-by-step solution generated by the model, the annotator needs to read the question and model-gencrated solutions carefully. There will be different solutions from multiple models for one question, and they need to be annotated independently. Each annotation task is divided into four sub-tasks.
2.1 Sub-task 1
The first sub-task is to decide whether the mroel--ennerted step-by-step solution is correct. Even if the model's final answer matches the ground-truth answer, the model may arrive at the correct answer through incorrect understanding
reasoning. Therefore, regardless of whether the final answers match the ground- truth choice, annotators should read through each model solution carefully and complete all subtasks based on prudent judgments.
2.2
Sub-task 2
The second sub-task is identify the first incorrect step in the model gener-ggne solution if the solution is incorrect. Specifically, each step can be interpreted as correct, neutral, or incorrect.
Correct means both the calculation and understanding are correct, and the step essential for reaching the correct result.
Neutral refers to the current step possibly being a background statement about question, a factual/formula description, or an exploratory step generated by the model. There are obvious errors in this step, but whether this step leads the final correct answer is not obvious.
Incorrect step refers to obvious errors in calculation, understanding, other possible emror reasoning, regardless of whether this error affects the final calculation result.
Please note that only need to find the first 'incorrect' step.
2.3
Sub-task 3
The third sub-task is to describe the reason for the error in the first incorrect step. Annotators should carefully analyze the possible reasons for the error in the incorrect step and provide rational analysis. Please try to incorporate the logic of the model's solution and analyze the reasons for this error from a holistic perspective, This error reason may the future, please be as detailed, comprehensive, and clear as possible.
reference for other annotators models in
59

---

2.4 Sub-task 4
The fourth and final sub-task rewrite the first incorrect step. Based on the error reason you provided, correct the incorrect step that you identified. The rewritten step should naturally integrate into the original context of the model solution, with only the incorrect part modified when necessary.
3 Annotation Data Format
the end of this guideline. Figure shows an annotation example of one model-generated solution for a logic question, which includes the following items:
Question, Options. Wrodel_Souution Steps and Ground Truth_Answer They are what annotators should carefully read and analyze.
Model_ Saaiee__Sottto_____ Corrrcttess is the first sub-task item to be annotated. It defined whether the model solution is completely correct for all steps. Please only answer "Yes" or "No" in this column. Please do not fill other variants such as correct, incorrect, positive, negative, etc.
Model Solution First Error_Stee Step Num is the second sub-task item to be annotated. It is defined as the first incorrect step specified in the model solution. If the annotation of the previous item is "No", please enter the numerical step number with quotation marks here, such as "1", "2", etc.; If the result of the first item annotation "Yes", in "None" here, Please note:
Step_Num
Do fill in other variants such as "one", "two", "none", "no", etc.
2. Step numbers should be written in double quotation marks; do not just write numhers.
Model Solution Error Reason is the third sub-task item to be annotated. Please provide a detailed error analysis for the identified first incorrect step here. For example, whether the model refers incorrect factual knowledge/ correct facts but incorrect comprehension, calculation incorrect contextual logic links / improper conditions or assumptions, etc. the annotation of the first item "Model SalpelSSoouuotSSSeeeeSSSCCC Correettesr"""""""""" is "Yes", fill in "None" here.
Rectified Model Solution First Error Step is the fourth and last sub-task item to be annotated. Please correct the incorrect step based on the error reason you provided. If the annotation of the first item "Model_Solution Soluution_Corrccttessst Correctness" is "Yes", fill "None" here. Note that we are not responsible for the consequential steps following the first incorrect step; we only need to correct the identified first error here.
4 Annotation Regulations & Quality Control
All questions are to anmotated according to the format requirements. If issues arise with the annotation format from sampling inspection, annotators should make modifications promptly.
If the annotations of error reasons and step corrections are too brief, perfunctory, or unclear, re- annotation is required to ensure the annotation quality. For each batch of submitted files, if any of the following occurred, all questions of the same batch will be sent back for re-cxamination and re-annotation:
1. the annotation error rate is higher than 5% after random sampling, or 2. any of the requirements in this guideline found violated.
5 Additional Notes for Logic Subject
5.1 Look-Up Source
Annotating the logic subject questions requires prudent reasoning and inference abilities, as anno- tators have been notified and assessed accordingly in the recruitment round. When encountering unfamiliar or unclear terms and concepts, please use search engines and conduct factual inquiries from authoritative sources to assist with your annotation. Please do not use any language models as
your annotation assistance tool. Contact the annotation management whenever in doubt.
60

---

6 Common Questions
Q1: When step repeats the incorrect option should mark it as the first incorrect step?
No, if step only describes what an incorrect option stating or repeats the exact words of the incorrect option without analysis, does not constitule an incorrect step. But if the step includes wrong unalysis/judgment about the incorrect option, then it considered incorrect.
Q2: If previous steps so far have only stated relevant facts without specific analysis, how should I handle these steps?
Similarly, if the steps focus solely on factual statements about the options or related background descriptions, these steps should be understood as neutral steps unless these steps contain incorrect understanding, incorrect facts, etc. The first incorrect step should be the one that involves incorrect analysis or reasoning.
Q3: What should I do if encounter difficulty comprehending the question options model solutions?
When encountering unfamiliar terms, look up textbook sources or use search engines to find reliable explanations first. Contact the annotation management team whenever in doubt.
Q4: The model's answer gives two answers, one of which is correct. How should decide?
Annolate as "No." Then follow the standard process to annotate incorrect step, error reason, and the correction step.
Q5: It's inconvenient read the JSON file without line breaks.
If you are using platforms like Visual Studio Code annolate files in JSON format, you install plugins like "json editor' to assist in reading. However, take caution against modifying any content other than the required subtasks and do not manually change the data structure.
7 Annotation Example
Figure below is model solution of sample question to annotated; Figure 2 shows an example annotation for the same model solution.
61

---

"queation, 20I0"; "ESSTESIe-Sub- C22 www------------- 16002224993 "Bilyeor" *togio", "Queatsen" "Seczation udocatien achools cnes poceobed daily comsupptio of Good fism sadh of Sour Fund groupar seat, "Optione";; "he healthy dist should Instude the consrngtion of sevecal differant festte and vagetablar delly, but reconmandation often interprated satiatied by the consungtion of zingle serving of fruis vegstable, val: The recoreandation that soea food from the fruit and vagstable croup be ceneored dasly conesituree
vegerables,
recemsnddetin sas,
dengerous health.Rsch
support
ranindes not to neclect this group, vhiet provides neede vitaking, ninerals, and fiber, 1ndi the oxisaion of fist, which contains benefielal olls, frmm the namsa of gcoupa in the List gave ecconsou iepesaiion that divisins inco dous guoupa qave the instuaiion that aqual excunt of each abould conzazad, but salk and zast sand awsunte. "
lass haiithy as food than red mett VaD: The
diseace coneer should eaten
"Ground Truth geiet **, "Geound Teath Amsretts "g",
"Heound teath Ansest Soditied's 107,
"Sangled Modal*; rclaude2",
"Model solutios Stepe"s
"step cheice provides support the oritique that recomennaatin was dangeroca to health, since Interpreting single frost or vegecable serving per day be rhheslchy, de chotes shoutd
Allowing conereption "*,
"Step 2: Choice a states the recommndeeioon served usstul peetnder contuse festoe and vegetables, whits promide hasith banetite, This supports the recomsnndrsion se rather than crteiquing the so Choice should alre be cliasnatedi,",
*Stap Cheice prevides critique the recomandation neting the criszion of Elsh, which baseticial fcod. supports the olats that the recomnendation use dangerous Tea' th.", *4,
"Siep 4: Carios D critiques the zscomsonitton noking that is implied equal consupption of Food gaoupa that abould vates lusuue anounta the to lmslith dangeza. This supporia the clean Lha, the cucomantation was dampezouaa.
*stup St choices A, 8 and all provide suport for Uhe ucllique of the cecomanart.ons vhile onl cheioe it does not."y
*dolctions choice
dototion Correntnese"i 1780s,
Model Solution Firat, Ercos Stephs "zac",
Nodel Jolotion Rrrer Reasents "9807,
"Ndde] 201 slioo twocilied Picss teane suiphs "I307,
Figure I: A lask example of a model-generated solution for a logic question (to be annotated).
_ooeltio_nninnn Dorrriteeeerrreeeeeee.e "incorrect", "Modal Solution First Frrt__Fro___rpp____ *2",
"Model MModee_SSouuee Solution Brror Reason"" "Option B aupports the importance of consuning fruits and vegetables in the diet. This contradiets the claim that this recomnendttinn is dangerous to health."
"Step Option supports the pact of the resemmezcation about consuming truits and vegetables, and does not provide critician ot entira recomnenaation. So option B ahould be correct anaver."
"Medell "Wedee__Sluttio" Ltteiiiiooeeeeeeieeedd__eeeeedde
Figure 2: An example of standard annotation (with the previous items omitted to save space).
62

---

Annotation Guideline: Coding Subject
1
Introduction
With the capabilities of large language models like ChatGPT gradually increasing, existing models often manage to decompose and partially solve complex problems. However, their explanations may sometimes be misleading contain errors in understanding, calculation, and logical reasoning. In this annotation task, annotators should review and ovaluate the step-by-step solutions provided by the model to determine whether the model has correctly understood and reasoned to obtain the correct answer.
2 Annotation Task Description
Given question and step-by-step solution generated by the model, annotators need carefully read the question and solution, and review the solution. Each question will have multiple different solutions from the model, which need to be reviewed independently.
2.1
Data Format
Each data entry will include:
A coding question and description (within block comment the function).
A segment of the solution generated by the model.
A series of assert statements to help check the generated code.
The comments in the model--eneeattel code solution should indicate the model's thought process. We should regard comments as steps in the thought process, and the code corresponding to each comment as the model's implementation of the comment. We can run the generated code and use assert statements to test and understand the code's 's logic to complete the following tasks.
2.2 Tasks
Task 1: Determine the correctness of the step-by-step thought process. Even if the model's final answer does not produce an error, it might have arrived at the correct answer through incorrect understanding and reasoning. Therefore, regardless of whether the final answer matches, annotators should carefully read the question and understand the code, using assert statements identify any errors.
Task 2: Identify the first erroneous step. If the solution is incorrect, annotators need to identify the first erroneous step in the step-by-step solution. Specifically, there a logical error in the code, find the first incorrect thought step's s comment. If the code logic is correct, but the implementation contains an error, identify the first line of erroneous code. If the model failed to correctly understand intent of the question, find the first step explaining the thought process. Note that we need to find the first erroneous step.
Task 3: Describe the reason for the first error. Annotators need to analyze the error in the erroneous step and provide reasonable analysis. Please try to incorporate the model's s thoughi process into the analysis. This error reason will serve reference for other annotators models in the future, so please be detailed, thorough, and easy to understand.
63

---

Task 4: Rewrite the erroneous step. Based on the provided error reason, correct the erroneous step. If the thought process is incorrect, modify the erroneous comment. If the code contains an error, modify the corresponding code. Note that we only need to correct the first line of erroneous code. If you have any questions about this step, please contact
3
Annotation File and Format Description
Below is sample annotation for code answer (in response model reply).
Model_Solute Saluutio__ttteeeee Solepi and assert_ teteerrsstttemettte are the content that annotators should carefully read, amalyze, and run tests on.
Model_Solution htt._oooottooo.oorreeeeeees Correctness is the first item to be annotated: please only answer "yes" or "no" in this column. Do not in other variants such as correct, incorrect, right, wrong, etc.
Sobel_Soluti on_ to___irr__Errr _erssrrrrr____ Sttrs_Step__ii _Nu is the second item to be annotated: if the first item annotation result is "no", then only answer the number of the erroncous code line in this column such as "1", "2". If the first item annotation result is "yes", leave this column blank. Note:
Do not fill in other variants such "one", "two" or "none", "no".
The step number should be written within quotation marks "" do write just the number. (Line numbers can be enabled in the compiler, for example, in VSCode notebook, you can open the command palette and find Show Cell Line Numbers)
The line number starts from the first line of the code block. If there is no code block symbol, start counting from the first line of import or def.
Model_Solute contion__aii Errerr_eoaRooo is the third item to be annotated: please fill in this column with a detailed eгror analysis. Explain in a three-step method: 1. What was the model's original solution trying do, and where did the problem occur. 2. Give example of why the model could not solve this step correctly. 3. Provide a potential correct solution thought, what should be done in this step. If the first item annotation result is "yes", leave this column blank.
led_iiied_Mha_ ied__tteeee ieddddddddddoe wtdeeoooodetiiinooo Swwlttionnliiiteeeeeeoe Fir___rst_____ Eerrss_EErrr__ee _tep is the fourth and last item to be anno- tated: in this column, rewrite the erroneous step correctly. Try to provide a correct step in the same style as the model so that can directly replace the original erroneous step. If the first item annotation result is "yes", leave this column blank, Note:
Rectif
We are responsible for the steps after the first error. We only need correct the
error here.
Each question may have multiple correct answers after the first error, we only need to choose one way modify it.
Please ensure the modificd code has correct indentation.
4 FAQ
Q: the code has multiple code blocks, how to locate the line number?
A: We have regenerated the model answers and ensured stricter format, now all code should within one code block. If you still find such cases, please contact us.
Q: What if the model answer is abnormal (e.g., code)? What if the question is incomprehensible, or the question itself seems problematic?
A: Please write the issue in the feedback sheet below. We will follow up on the problem.
3. Q: If the generated code lacks library and function, what should be the error line number?
A: All are considered error.. The error step should be located below the earliest existing import line, rather than where the inner function/library function is used.
64

---

5 Annotation Example
5.1 Items to Read / Run
5.1.1
Model_
def divide_string(s: str) ->bool:
A string composed entirely of repetitive characters is known uniform string.
For example, "1111" and "33" uniform strings. contrast, "123" is not uniform string.
Rule: Given numeric string 8, divide string into some uniform substrings, such that
there exactly uniform substring of length and all other uniform length of 3.
substrings have
If the string otherwise, return false.
divided according rule mentioned above, return true;
Substrings are sequences of contiguous characters within the original atring. >>> Lividd__string("001111000
false
>>> true
Check if length of a is valid if (sn 3 1=2: return False
1 =
while 1 <len(s):
# Find substring of length 2 if < <len (s) -2 and s[i] -ms[i+1]:
substr_len =2
Find substrings of length 3 else:
substr_ =3
Check if current subatring is uniform
subetr = [1:iissbbtr__ee _len] if len !=1: return False
+- substt_lenm
True
The key steps are:
Check if length of is valid based on the constraints.
Iterate through s.
3. Al each position, try to extract substring of length 2 or 3.
4. Check if the substring is uniform.
Return False if any substring not uniform.
If all substrings are valid, return True at the end.
The comments explain the reasoning and logic behind each step.
65

---

5.1.2
assert_statementte:
assert dividd.strrnn((000111000))) --False
assert (ivdeeestrinng((001111222)) =-True
assert divie__ernn((0110022233)) ==False
divide_strann( ==False
divide_string (trrierstrrng(4443344)4))
=-True
5.2
Items to Annotate
5.2.1
'no'
5.2.2
'23'
5.2.3
"The model tries extract substring of length and but the 'if' statement will skip the possibility finding any uniform substring of length 3. For example, if the input is '00011', ?, the first uniform substring '000* will be incorrectly split at '00'. One possible solution is to record the length of all uniform substrings and check if their lengths satisfy the requirements. Therefore, at this step the model should find the length of the current uniform substring."
5,2.4 Rectified Rectiffed_Model, Moweeoodeeooutioonoiiiii..ee Brrrr Step:
Find the length of the uniform substring
66

---

Three-Round Interaction Prompt Template for Self-Refine
Following is a question/solution pair in subject {sol[ "Subjecct J}. Your task is to examine the solutions step by step and determine the solution correctness. If the solution is incorrect, please further find out the first error step and explain the error reason.
[Generated Response From Evaluated Model)
Review your previous answer and find problems with your answer
(Review Response From Evaluated Model)
Based on the problems you found, improve your answer.
Please follow the desired response format:
(Self-Refined Response From Evaluated Model}
Figure 8: This is the prompt we used for self-refine experiment, note that three consecutive inference calls are made in order to perform the most basic self correction.
Task 1 GPI-4-turbo
Task Liama 3-708- 7078-Irstruct
Task Liama (attma----B---eee BB-Instruct
Task gemma-2bitt
8.56
14.%
4.7%..%
22.3%
22.8%
27.2%
35.7%
Task 2 GPT-4-turbo
Task 2 Llama-3-708-Ineeee 10--Instruct
Task 2 1llmma-3-8B-Instien
Task 2 gamma-2b-lt
48%
7.0%5
15.0%
10.1%
16.94
SCUN
A%
11.3
95.1% Task gemma 2b a0Ri%
Task 3 GPT-4-tuur turbb
Task Liama 3-70B Instruct
Task 3 Liama 3 8B Instruct
0.2%
11.%
a.d 226
15.5%
13.9%
21.5%
6311
76.6%
93.0%
Incoerect
Figure 9: This is the performance breakdown for self-refine experiment in the task level, where task 1,2,3 refers solution correctness, first error step and error reason determination.
72

---

Prompt for Response Generation
Following is question/salltion pair in subject [sal[ ("Subject'))- Your task is to examine the solutions step step and deternine solution correctness. If the solution is incorrect, please further find out the first error step and explain the enror reason.
Following are the specific definitions of the fiels:
Solution corractness: Does the solution correctly the corrected options?
question with justifiable reasoning and selected
First Error Step: For
step can either be correct, neutral Incorrect. Correct steps are those that answer.
corputation Neutral steps are those step that are explanatory, exploring or focusing on background illustration. They very clear if they lead Incorrect steps are those with factual errors, conputation errars or lnderstanding/looit arrors. These steps path incorrect need single out the first step that comes with above errors or lead to incorrect answers. Error Reason: For the identified first error step, please specify the errors nade in this step and suggest rectified reasoning stop instead.
possess sound logic
but
night night not detour the
(k_ shoth deno)
Below is the question and solution for you to solve:
Question: [sol["Questte 'question
options: (sal ('Options') }]}
Step by Solution: [Mde[ solution_Steps"
(hint_sent)
follow
response forat:
Solution Analysis: [Give step by step analysis an the solution correctness here]
Solution Correctness: [Input 'correct' i'icorrect''
to First Error Step: [Input 'Step x' here to indicate the first error step here. Input 'N/A" if the solution is correct.]
correctness of solution]
Error Reason: [Input the error reason and the rectified reasoning of the first error step here. Input 'N/A' the solution correct.]
Please follow this fornat without any additional introductory concluding statenents.
Figure 10:
This is the prompt template used evaluate all the models. The k-shot-demo and hint-sent are either the few shot examples and solution correctness prior or empty string, depending on the experiment setup.
73

---

Prompt for Scoring Error Reasons
teacher, your assistance required to evaluate a student's explanation regarding the error in a problem solution. The task involves a detailed understanding the problem, the incorrect solution provided, and the ground truth behind the error. Your analysis should focus on whether the student's explanation aligns with actual error in the solution.
As an experienced
Please find the details below:
Question: [data[ Question']]
Incorrect Solution Provided: {data[" 'Model "oodel Solution Steps' S
First Incorrect Step in the Solution: {data[ "Model. Solution Firs___it____eeee
Biitn__srssttrrrstt_aearrsr Error_ Eettt__rrrr__p }]
Ground Truth Ennor Reasons: {data MModel_Solution Erronn__rr____er
____er_Reeeoonn_ Reason']]
Ground Truth Rectified Steps: (Madal LaaoooelMoouuttiiiieeee Solution_Rectifiie First_Erron Step' ']}
Student's 's Explanation of the Enron: {data("Evaluse (evvlllaaiionnnnnneaaa Result eeeeroorreeaoon }]]
Based this information, please provide the following:
1. Step- Reasoning: [Offer succinct, step-by-step interpretation of the ground truth reason.]
2. Student Error Reason Analysis: [Analyze the student's explanation step step, determining its accuracy in reflecting the actual error briefly.]
3. Final Decision: [State only 'Correct' "Wrong" to conclude whether the student's s explanation correctly identifies the error based your analysis.]
Please follow this format without additional introductory or concluding statements.
Figure 11: This is the prompt template used to request GPT-4 to help explained by evaluated models. Note that despite the difficulties of deciding the solution correctness, it is much easier decide the error reason correctness given the ground truth annotations.
the enror reasons
74

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of this work are discussed in $ A.1
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
75

---

Justification:
Our work does not involve theoretical assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all the information needed to reproduce the main experimental results the paper to the extent that it affects the main claims and conclusions of the paper, detailed in $ 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
76

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We opensourced evaluation benchmark and the script described in Section-]
Additionally, we have detailed the experimental setup in the paper ($ ( including model selection, hyperparameter settings, data selection, evaluation metrics, hardware resources, etc.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
public/guude//Cdeeuuuueee bosdddSSubmsssioooooccy.o for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurIPS code and data submission guidelines (https / ctltsps.ccpubbicc/// gwtcc//gugeggeee sonneeSSumissiioPPooe
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide comprehensive dataset statistics, evaluation metric descriptions, hyperparameters, and tool usage in
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our proposed method is an inference-only approach for LLM and we adopt the decodddg strategy for all of our experiments, making the experiment results of each session consistent.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
77

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer:[Yes,
We provide comprehensive experimental setup and hardware computation resources used in
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:://eerrips.ccppbiit////teee
Answer: [Yes]
Justification: We confirm that the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics, and all the authors preserve anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts of our paper are presented in A.2
Guidelines:
The answer NA means that there is no societal impact of the work performed.
78

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our dataset focuses on evaluation rather than training models. We leverage existing datasets rather than scrape from the Internet.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the assets, i.e., codes, data and models used in our paper, are properly credited and we explicitly mention and properly respect the license and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
79

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have submitted the anonymized dataset, codes, and corresponding docu- ments together with the paper.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
The full text of instructions given to human annotators is presented in H
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The justification as follows: We solely engaged human annotators for the dataset, and they not subjects of our study. Furthermore, partnered with legally recognized annotation company in the country, which has obtained all necessary governmental approvals to operate its annotation business.
Guidelines:
80

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
81


Reviews:
[{"review_id": "YjTtuMmx5F", "reviewer": "Reviewer_SioJ", "summary": "This paper proposes a meta-reasoning benchmark for evaluating the solutions generated by a large-language model (LLM) to shift the focus more to process-based evaluation of an LLM's reasoning abilities rather than outcome-based evaluation.\n\n*Evaluation*:\nOn a variety of question-solution pairs, the model is asked to score its own generated solution for correctness, to identify the first erroneous step, and if these are correctly identified, the reasoning of it solution is evaluated by GPT-4. These three are used to compute three metrics respectively-- the MCC for a binary classification of the solution correctness, the ACC_step to judge the number of correctly predicted first-error steps as compared to the total number of incorrect solutions, and the ACC_reason to judge the number of correctly predicted first-error steps and error-reasons as compared to the total number of incorrect solutions. These three are combined to form an MR-score that is used to compare different models.\n\n*Dataset*:\nThe question-solution pairs (~6k in all) themselves are formed such that the questions are sampled from MMLU (arithmetic reasoning), LogiQA(logical reasoning), and MHPP (code-based reasoning) and the solutions are generated via chain-of-thought prompting by GPT-3.5-Turbo-0125, Claude-2, and Mistral-Medium. The annotations for these question-solution pairs are annotated by human annotators-- these annotations of solution correctness, first error-step, and the error-reason are used for evaluation.", "questions": "Some questions for clarification:\n- I might have missed this, so in Table 2: is the meaning of k the number of shots of prompting the model gets? If so, it would help to mention that in the caption since k is not mentioned in the text anywhere else.\nwhat should be the take-away on self-correction?\n- Is the difference between the Self-Refine prompting (section 6.2) and Solution Correctness Prior (section 6.3) the absence/presence of of the ground truth (resp.)? How do the two compare with each other?", "limitations": "- The proposed benchmark also relies on step-wise solutions to evaluate reasoning which reduces its applicability and novelty as compared to existing work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors highlight the importance of more carefully evaluating the solutions generated by LLMs to complex problems rather than just comparing their accuracy which is a notable contribution to the body of work of improving evaluation of LLMs.\n- The annotation process followed for experimental evaluation is thorough and multi-step for quality assurance\n- The authors raise an interesting question about problem solving dynamics, which can be potentially explored further in future work on LLMs", "weaknesses": "These questions/remarks might also have arisen due to my lack of proper understanding, so I am willing to increase my score if these can be clarified:\n- My main issue is with the mixing of reasoning based and accuracy based evaluation into a singular score. What is the reasoning behind combining the three metrics (MCC, ACC_step, and ACC_reason) into a single score (MR-Score)? Does this not affect the motivating insight that models might have a high MCC but a super low ACC_step and an even lower ACC_reason? Is there somewhere the three metrics can be viewed individually also so it's clearer how the models vary along the three-- right now, it is not clear to me by looking at the MR-scores in Table 2 how the 15 models actually fare when it comes to identifying the correct error-reason, especially when the MR-scores for the models are in the same ballpark.\n\nWriting:\n- (pre-existing work): it's not clear to me what the real differentiating aspect of MR. Bean is vis-a-vis MR-GSM8K and MR-Math. MR-GSM8K and MR-Math also consider the first-error reason? So, is it that they are math-reasoning based, and MR. Bean now just combines multiple datasets to essentially follow the same evaluation pipeline? If so, this should be clearly mentioned. They don't just go a step further, that is the furthest step gone to, even considering this work (if that is indeed the case).\n- The abstract makes it seem like the MR-score is a metric designed by the authors in this work, although it's already been proposed \"*Through our designed metrics*\" which is somewhat over-selling the work.\n- (minor) typo: A.2 Negative Societal Impacts\n- The connection between task difficulty and reasoning capability can be explored further-- I am assuming ACC_reason on more difficult tasks would be comparably worse than the MCC, which would be the actual assessment of reasoning capabilities. Also, from Fig 2, it seems that Logic based tasks are the most difficult, then wouldn't it be fair to include them in the comparison too or perform the comparison of task difficulty not just based on high-school/college, but also take into consideration how the model actually interprets the task difficulty based on MR-score?\n\nExperiments:\n- what is the main takeaway on self-refine prompting since it seems to be in contrast to existing observations? How does the shift from incorrect to correct predictions depend on the model family and task? Is it seen only across MR-Scores or also in the ACC_reason (which should probably be affected more by model size?)? \n- Is the difficulty of logic-based reasoning also because they contain the longest questions? is there any impact of question length on the model's reasoning ability, if not solution length?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:\n\n> **W1: Mixing metrics into a singular score**\n\nGiven the **interdependent and progressive** nature of the three tasks (MCC - ACC_step - ACC_reason), we can either choose to combine them **organically** by assigning weights that **consider both the differentiability and interpretability,** or simply report the final metric ACC_reason. Considering the three metrics are complementary in revealing how models perform in the respective dimension as you also suggested, we claim the necessity for MR-Scores as follows:\n\n- **Unified Metric**: The MR-Score offers a unified and normalized metric that balances the difficulty levels of the three sub-tasks (lines 198-199).  We conducted a thorough grid search to determine the weights for the sub-metrics and found that the MR-Scores are insensitive to relatively minor (e.g. ~0.1) adjustments on the weightings. We therefore chose the schema that assigned the greatest weight to ACC_reason and the least weight to MCC, as error reason reflects explicitly the model's understanding of the rationale behind the question. We believe **the current weighting ratio strikes a good balance between interpretability and differentiation**: For example GPT-4-Turbo, Deepseek-v2-236B and Mistral-Large achieve 86.4% ,78.5% and 81.2% respectively in MMLU but score 43.2%, 29.4% and 21.3% in our benchmark. \n\n- **MCC and its correlations**: MCC is chosen because it effectively penalizes random or biased behaviors. Although given the progressive natures of the three tasks, the raw scores of MCC, ACC_step, and ACC_reason are generally in diminishing order, the correlations are not too high (corr(MCC vs ACC_step)=0.42, corr(MCC vs ACC_reason)=0.46). Therefore, all three metrics are necessary components to provide an accurate and thorough evaluation.  \n\n- **Individual metrics**: Due to space limit, **we report the sub-table into the pdf posted under the global response section of this rebuttal**. \n\n> **W2:  Differences among MR.Bean, MR-GSM8K and MR-Math.** \n\nOur work builds upon previous efforts and introduces several key improvements.\n- **Extensive domain coverage**: By recruiting and training a diverse group of domain experts as annotators and operating meticulous supervision and quality control, we extended MR.Bean to be a comprehensive benchmark covering coding, logic, medicine, and science, in addition to math, providing a broader evaluation spectrum. \n\n- **Larger scale and increased difficulty**: While MR-GSM8K and MR-Math focus on primary and high school math competition, MR.Bean increased the difficulty to high school to graduate level. MR-GSM8K and MR-Math are both limited in dataset size, **while MR.Bean is 199.1% and 1,195% larger**, respectively.  The scaling up and difficulty diversity both contribute to a more robust evaluation. \n\n- **Rigorous and fine-grained annotations**:  MR-Math only considered solution correctness and the first error step. MR-GSM8K additionally annotated the first error reason. However, each problem in MR-GSM8K only contains annotations of a single solution and a single error reason. In MR.Bean, **each problem is mapped to three solutions sampled from different SOTA models and the error reasons are provided by multiple annotators who agreed on solution correctness and the first error step. Additionally, we annotate the revisions of the first error step.** The revisions are ultimately integrated into the error reasons used by proxy LLMs as a reference to score the error reasons generated by evaluating models.\n\n\n> **W3: Length and difficulties**\n\nThe table below shows the zero-shot average MR-Scores of SOTA models per subject sorted in the ascending order of question length (measured by the number of words). Other question statistics can be found in Table 1 of the paper. We can indeed observe a negative correlation between the performance and the question length. \n\n| Model          | Math  | Chemistry | Biology | Physics | Medicine | Coding | Logic |\n|----------------|-------|-----------|---------|---------|----------|--------|-------|\n| (question length) | 44.3 | 48.1      | 56.3    | 66.6    | 88.7     | 140.1  | 154.8 |\n| mistral-large  | 21.53 | 24.49     | 21.48   | 24.27   | 16.34    | 21.8   | 15.1  |\n| deepseek-chat  | 32.18 | 32.52     | 29.97   | 32.44   | 26.54    | 34.18  | 23.58 |\n| gpt-4-turbo    | 44.28 | 41.71     | 44.77   | 42.54   | 38.89    | 50.99  | 30.98 |  \n\nFor further quantitative investigations, we conducted a Pearson correlation analysis:\n- **Question Length**: We found a high Pearson correlation coefficient of -0.58 between question length and model performance, indicating a relatively strong correlation.\n- **Solution Length**: The correlation coefficient is -0.29, showing a lesser but still notable effect.\n\nHowever, it is important to note that the difficulty of the subject question is a complicated factor and not solely dependent on the question length. For example, the difficulty of logic questions extends beyond their length to their inherent abstractness and the need for commonsense and real-world understanding, since its questions are sourced from the LogiQA dataset, originally collected from the Civil Service Entrance Exam (for case demonstrations, please refer to Appendix E-6).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ccy1lBdGLY", "reviewer": "Reviewer_TVR6", "summary": "This paper introduces MR.BEAN, a comprehensive benchmark for evaluating meta-reasoning capabilities of large language models (LLMs). Comprising 6,006 questions across various subjects including physics, chemistry, logic, coding, and more, MR.BEAN requires LLMs to analyze and correct errors in automatically generated reasoning steps. The benchmark is meticulously constructed using a three-step annotation process involving answer correctness evaluation, error step identification, and error reason analysis. \n\nThe main contributions of this work are:\n1. A novel, large-scale benchmark for meta-reasoning evaluation covering diverse subjects and reasoning types.\n2. A rigorous methodology for creating and annotating meta-reasoning questions, ensuring high-quality data.\n3. Comprehensive evaluation of 15 LLMs on the benchmark, revealing limitations and weaknesses in their reasoning abilities.", "questions": "1. Can you provide more details on the annotation process, including inter-annotator agreement rates and resolution of disagreements?\n2. What steps were taken to ensure that MR.BEAN specifically measures meta-reasoning abilities and not merely language understanding or domain knowledge? \n3. How sensitive is the dataset to the specific prompts for generating solutions and errors? Did you experiment with different prompt formulations, and if so, how did this affect the resulting dataset?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Comprehensive and well-organized dataset: MR.BEAN covers many subjects (p.3, Table 1) and offers a broad assessment of LLM meta-reasoning capabilities across diverse domains. The paper is structured clearly, with detailed explanations of the dataset creation process, evaluation metrics, and experimental results.\n2. Novel meta-reasoning focus: By requiring LLMs to identify and correct errors in reasoning, MR.BEAN offers a unique perspective on evaluating AI reasoning capabilities, going beyond traditional outcome-based assessments.\n3. Extensive empirical study and analysis: The authors evaluate different LLMs on MR.BEAN and have tested different prompting methods to comprehensively analyze current model capabilities and reveal interesting limitations in their reasoning abilities.", "weaknesses": "1. Validation of meta-reasoning specificity: While the paper describes the evaluation metrics in detail (p.5, l.185-197), it is still not clear such metrics measures meta-reasoning abilities rather than general language understanding or domain knowledge.\n2. Prompt sensitivity: The paper doesn't adequately address the potential impact of prompt design on the generated solutions and errors. Given that LLMs are known to be sensitive to prompt wording, this could significantly affect the nature and distribution of errors in the dataset. \n3. Lack of annotation details: While the paper mentions a three-stage annotation process (p.4, l.116-135), it lacks specific details on annotator qualifications, training, and inter-annotator agreement rates. This information is crucial for assessing the reliability and consistency of the annotations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind review and insightful and to-the-point comments. We are committed to addressing your concerns and providing clarifications.\n\n> **W1: How does our evaluation mechanism measure meta-reasoning abilities rather than general language understanding or domain knowledge**\n\nWe believe language understanding and domain knowledge are inseparable and essential components of reasoning. Examples include MMLU, LogiQA, and MHPP. These well-recognized reasoning evaluation benchmarks require substantial language understanding ability and application of domain knowledge. However, as outlined in our abstract and introduction, **these benchmarks primarily use a result-oriented evaluation method rather than a process-oriented one**. This can be flawed due to incorrect understanding and reasoning.\n\nOur benchmark addresses this limitation by proposing a meta-reasoning framework that transforms evaluating models from the role of students generating results to the role of teachers scoring solution processes. **To effectively score these processes, evaluating models must actively reflect on and criticize conditions, assumptions, and logic, and examine potential outcomes counterfactually.** The above capabilities are essential for a more robust and trustworthy reasoning mechanism. \n\nTherefore, our meta-reasoning paradigm provides a challenging yet feasible evaluation pipeline that forces models to \"reason about reasoning\" (i.e., score candidate solutions), which we term meta-reasoning. This paradigm aims to evaluate the reasoning in a more fine-grained approach (measured in a series of subtasks and represented under a unified metric), and to do so it indeed requires advanced language understanding and mastery of domain knowledge as you suggested.\n\n> **W2: The paper doesn't adequately address the potential impact of the prompt.**\n\nWe agree that language models can be susceptible to different prompting methods. \n- **Response generation prompt design**:  We fully agree with you that language models are susceptible to prompt wordings and sometimes it can affect the distribution of reasoning errors in the evaluation benchmark. With this in mind, **our design of response generation prompt are following the general best practices of prompt engineering guidelines.** As illustrated in Figure-10 in the Appendix, our prompt has the following key attributes: (1) Clear task description and background information, (2) Persona adoption as an experience scoring teacher, (3) Divide-and-conquer fashion that splits the goal into sub-tasks, (4) Allowing “time” for the model to think in step by step manner, (5) Clear separation of different parts of information, and (6) Specific requirements on the return format.\n\nThe final prompt presented in Figure-10 is actually the iterated version after considering the prompt wording effects and is proven effective as explained in lines 219-222. \n\n- **Few-Shot In-Context Learning**: In section 6.1, we experimented with this method and observed performance fluctuations across models of different sizes. We suspect lengthy demonstrations might confuse the models. To prove our point that length does affect the model performance, we conducted a Pearson correlation analysis. In the results, we found a Pearson correlation coefficient of -0.58 between model performance and question length and -0.29 on solution length. This negative correlation supports our hypothesis above.  \n- **Self-Reflect Prompting Method**: In section 6.2, we adopted a response-examine-refine pattern. This did not significantly boost performance, as models often switched their decisions from correct to incorrect and vice versa.\n- **Ground Truth Solution Correctness**: We provided hints of ground truth solution correctness to see if models could better identify error steps and reasons. The positive results indicate that correct prior knowledge in the prompt can indeed affect outcomes.\n\nOverall, while prompting methods can influence performance, **our benchmark provides a robust evaluation mechanism that remains relatively stable** unless influenced by explicit hints.\n\n\n\n> **W3: The paper lacks specific details on annotator qualifications, training, and inter-annotator agreement rates.**\n\nBased on your suggestion, we provide the annotation details as follows. We are committed to including these details in the revised version of the paper.\n\n- **Annotator Qualification & Training**: As mentioned in lines 152-157, our annotators hold a minimum of a bachelor's degree. Each annotator is required to read through the annotation guidelines listed in Appendix G before completing a trial labeling process. The selection of annotators is based on their performance on a balanced small hold-out set of problems for each subject. \n- **Initial Annotation**: As mentioned in section 3.4, each question was labeled by two different annotators. Inconsistencies in solution correctness or the first error step were identified and reviewed by a quality controller for arbitration.\n- **Disagreement Resolution**: We did not adopt majority voting due to the objective nature of our questions. Instead, a senior supervisor reviewed all questions with disagreements to resolve ambiguities (therefore the inter-annotator agreement rate is not applicable here).\n- **Quality Control**: In the final quality control phase, 10% of the problems were randomly sampled and reviewed by meta-controllers (authors). The author-annotator agreement rate had to exceed 90% for annotations to be accepted.\n- **Annotator Details**: Each subject usually comprised 5-6 annotators with two project supervisors for quality control. The annotation process took approximately three weeks, followed by two weeks for quality control and resolving disagreements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1CVuVI5to3", "reviewer": "Reviewer_AKte", "summary": "This paper introduces a new benchmark for evaluating the reasoning capabilities of large language models (LLMs). Current methods primarily focus on final outcomes and not sufficiently capture the intricacies of the reasoning process. To address this issue, this paper propose MR.BEAN, a process-based benchmark that demands meta-reasoning skills from LLMs, requiring models to identify and analyze potential errors in automatically generated reasoning steps.\n\nThe authors conducted an extensive analysis of various LLMs using MR.BEAN, revealing significant limitations and previously unidentified weaknesses in their reasoning abilities. They found that while many LLMs can generate correct answers, they struggle to pinpoint and correct errors in the reasoning process. The paper also discusses the potential for improving reasoning abilities through techniques like the use of high-quality synthetic data.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper introduces a novel benchmark, MR.BEAN, which focuses on meta reasoning—a higher-order thinking skill. This design pushes beyond traditional outcome-based evaluations to assess the reasoning process itself. And MR.BEAN covers a wide range of subjects, including physics, chemistry, logic, coding, and more. \n2) The benchmark's questions and error analyses are curated and annotated by human experts, ensuring a high level of quality and relevance in the evaluation process.\n3) The paper's evaluation of various LLMs reveals previously unidentified weaknesses in their reasoning abilities, providing valuable insights for researchers and developers. The benchmark's application to a diverse array of models, from small to large, open-source and closed-source, provides a broad comparative analysis that can inform future development in AI reasoning.", "weaknesses": "1) The paper may not provide a thorough comparison between the model's automatic annotations and human annotations. Without such validation, it is challenging to assess the reliability and accuracy of the model-generated annotations.\n2) The concept of meta-reasoning typically involves instructing models on how to reason, which may include decision-making during the reasoning process. The paper primarily analyzes the reasoning steps after they have been generated, which might not fully align with the proactive aspect of meta-reasoning.\n3) The paper could provide more transparency regarding the annotation process, such as the total number of annotators involved, the total time spent on annotations, and the average number of samples annotated per person per day. This information is crucial for understanding the scalability and efficiency of the annotation process.\n3) The paper may not fully address how the dataset can be used to validate an LLM's reasoning capabilities on new, unseen examples. Understanding the generalizability of the findings is essential for assessing the true breadth of an LLM's reasoning skills.\n4) The paper could benefit from including specific test cases using GPT-4 to demonstrate the benchmark's effectiveness in identifying strengths and weaknesses in state-of-the-art models. This would provide a clearer picture of how GPT-4 performs against the benchmark and highlight the benchmark's utility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind review and insightful comments. We are committed to addressing your concerns and providing clarifications.\n\n> **W1: Missing a thorough comparison between the model's automatic annotations and human annotations.**\n\n(Since all the instances in MR. Bean are annotated manually, we hereby assume that you are referring to the automatic scoring of error reasons generated by the evaluating models)\n\nThank you for highlighting the importance of manual examination compared to automatic evaluations. We fully agree that manual examination can often provide a higher level of trust and thoroughness. However, we have designed our evaluation mechanism to provide robust and reliable automatic annotations of the model responses. \n\n- **Multiple annotators**: During the annotation stage, we collected multiple annotations for the first error reasons and potential error rectification from different annotators who agreed on the solution's correctness and the first error step.\n\n- **Proxy Model Evaluation**: Based on the ground truth annotations collected from various perspectives, the proxy language model (e.g., GPT-4-Turbo) then examines the error reasons provided by evaluating models. Given the question/solution pair and information regarding the first error step, error reasons, and rectification, the potential flaws of the error reasons provided by the evaluating models will be easy to diagnose under contrast.\n\n- **Agreement Rate**: As mentioned in lines 207-209, the agreement rate between manual annotations and the GPT-4 predictions over 100 samples randomly collected from all subjects is **92%**. This high agreement rate supports the reliability of our evaluation and therefore avoids the manual annotation of potentially 138,000 problems (6,000 benchmark sizes times 23 models evaluated).\n\n> **W2: Not aligned with the proactive aspect of meta-reasoning.**\n\nYour observation about the proactive nature of meta-reasoning is insightful. Since meta-reasoning is not yet a well-defined and recognized concept, we follow the definition of MR-GSM8K and MR-MATH to define meta-reasoning as an evaluation process that scores the reasoning steps. This definition, which tasks the evaluating models to reason about different reasonings, is different from the proactive prompting instruction methods in the following ways: \n\n- **Evaluation Mechanism**:  Even though the language models might adopt a more systematic and structured reasoning paradigm to answer the question, according to the proactive definition of the meta-reasoning paradigm you suggest, its effect is measured by the metrics that examine **the computation results only**. The quality and correctness of the intermediate reasoning steps are in no way guaranteed or reflected in the final metrics numbers. On the contrary, our evaluation mechanism provides an effective way that dissect the reasoning process into meticulously annotated subtasks, revealing the key properties of meta-reasoning ability.\n\n- **Teacher Role**: By shifting the model's role from a student generating answers to a teacher scoring solutions, **our mechanism forces the model to actively reflect on and critique conditions, assumptions, and logic, examining potential outcomes counterfactually.** All of the above are essential for a more robust reasoning model. In some sense, our definition of meta-reasoning is orthogonal and complementary to the proactive reasoning paradigm you suggested.\n\n\n> **W3: Annotation details**\n\nBased on your suggestion, we provide the annotation details as follows. We are committed to include these details in the revised version of the paper.\n\n- **Annotator Qualification & Training**: As mentioned in lines 152-157, our annotators hold a minimum of a bachelor's degree. Each annotator is required to read through the annotation guidelines listed in Appendix G before completing a trial labeling process. The selection of annotators is based on their performance on a balanced small hold-out set of problems for each subject. Every annotator is ensured to be paid above the local minimum wage rate.\n- **Initial Annotation**: As mentioned in section 3.4, each question was labeled by two different annotators. Inconsistencies in solution correctness or the first error step were identified and reviewed by a quality controller for arbitration.\n- **Quality Control**: In the final quality control phase, 10% of the problems were randomly sampled and reviewed by meta controllers (authors). The author-annotator agreement rate had to exceed 90% for annotations to be accepted.\n- **Annotator Details**: Each subject usually comprised 5-6 annotators with two project supervisors for quality control. The annotation process took approximately three weeks, followed by two weeks for quality control and resolving disagreements. Annotators generally handled around 20-30 questions per day, though this varied slightly depending on the difficulty level of the subject matter.\n\n\n> **W4: Validate an LLM's reasoning capabilities on new, unseen examples.** \n\nWe would like to clarify that one of the core novelties and contributions of our meta-reasoning paradigm is that someone can apply it to transform any “student answering” benchmark to a “teacher scoring” benchmark. **By successfully applying our benchmark on top of the well-recognized but already performance-saturated benchmarks like MMLU, LogiQA, and MHPP, we have observed a substantial performance drop for SOTA models** (e.g. Mistral-Large achieved ~80% accuracy in MMLU but scored 21.3 in our benchmark). This supports our point that the meta-reasoning evaluation pipeline is a challenging mechanism that uncovers the holisticness and comprehensiveness of language models regarding the mastery of domain knowledge and its application. We believe this paradigm will significantly contribute to the community, whether applied to existing datasets or new compilations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4brfZsD3Ti", "reviewer": "Reviewer_JijF", "summary": "The paper introduces MR.BEAN, a benchmark designed to evaluate the meta-reasoning capabilities of large language models (LLMs). This benchmark focuses on the models' ability to detect and correct errors in reasoning steps, addressing the limitations of existing outcome-based benchmarks.", "questions": "- Since the MR metric is newly proposed, would it be reasonable to incorporate human evaluations to support its credibility and relevance? By comparing the model outputs against human judgments on at least a sampled subset, it can provide empirical evidence on how well the MR metric aligns with human reasoning and offer insights into the absolute performance of models.", "limitations": "n/a", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-articulated, providing clear examples that effectively demonstrate the dataset's composition and intended utility.\n- This work contributes a new benchmark for identifying and correcting errors in reasoning steps rather than just the final outcomes.\n- The benchmark covers diverse topics and is paired with reasonable metrics.\n- A comprehensive set of experiments is conducted that can tell performance disparities among various LLMs.", "weaknesses": "- The $ACC_{reason}$ metric shows limitations in consistency. Its dependency on the judgments of different LLMs or human evaluators could lead to variability in scoring. \n- The weights assigned to different metrics might need recalibration when new models are tested or when the validation set is updated. \n- A detailed report on each metric's individual performance and impact is lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your kind review and insightful questions. We are more than happy to address them as follows:\n\n> **W1: The ACC_reason metric’s dependency on the judgments of different LLMs or human evaluators could lead to variability in scoring**\n\n\nWe would like to argue that due to the careful design of our evaluation mechanism, the automatic scoring of error reasons is both robust and economically feasible:\n\n- **Multiple annotators**: During the annotation stage, we collected multiple annotations for the first error reasons and potential error rectification from different annotators who agreed on the solution's correctness and the first error step.\n\n- **Proxy Model Evaluation**: Based on the ground truth **annotations collected from various perspectives**, the proxy language model (e.g., GPT-4-Turbo) then examines the error reasons provided by evaluating models. Given the question/solution pair and information regarding the first error step, error reasons, and rectification, **the potential flaws of the error reasons provided by the evaluating models will be easy to diagnose under contrast**.\n\n- **ACC_reason robustness**: Below is the scoring of error reasons sampled from our evaluation results. For the same set of error reasons collected in each subject, three different models made their predictions on the correctness/incorrectness. We can clearly see **the consistency of their predictions among three models over questions in all subjects**. Since the MR-Score is a weighted metric, the final score variability is less than 1 percent in total.\n\n| Model        | Coding | Physics | Biology | Math | Medicine | Chemistry | Logic |\n|--------------|--------|---------|---------|------|----------|-----------|-------|\n| gpt-4-turbo  | 83/55  | 137/15  | 164/11  | 305/46 | 194/25   | 166/27    | 192/16 |\n| deepseek_coder | 100/38 | 145/7   | 167/8   | 321/30 | 200/19   | 172/21    | 193/15 |\n| Qwen2-72B    | 99/39  | 142/10  | 167/8   | 312/39 | 195/24   | 172/21    | 200/8  |\n\n\n- **Agreement Rate**: As mentioned in lines 207-209, **the agreement rate between manual annotations and the GPT-4 predictions over 100 samples randomly collected from all subjects is 92%.** This high agreement rate also supports the reliability of our evaluation and therefore avoids the manual annotation of potentially 138,000 problems (6,000 benchmark sizes times 23 models evaluated).\n\n\n\n> **W2: The weights assigned to different metrics might need recalibration.**\n\nWe would like to clarify that, for consistency consideration, the weights assigned to different sub-metrics are not supposed to be recalibrated in the future, even when new models are tested. This is because:  \n\n**The discriminative ability of final MR-Scores is not sensitive to the weights**: We performed a comprehensive grid search over 23 models we evaluated. The results have shown that even for the large model coverage, the variance of MR-Scores across different models (representing the differentiability of MR-Score) does not change very much for different combinations of weights. We therefore considered both the difficulty levels of all three subtasks and their progressive nature, and selected the weighting schema that assigned increasing weights to solution correctness prediction, first error step determination, and error reason explanation.\n\n**The current weighting ratio strikes a good balance between interpretability and differentiation**: The traditional reasoning accuracy assigns similar scores to the SOTA LLMs, for example, GPT-4-Turbo, Deepseek-v2-236B, and Mistral-Large achieve 86.4%, 78.5% and 81.2% respectively in MMLU but score 43.2%, 29.4% and 21.3% in our benchmark. The increased difference in performance showcases our superior differentiability. \n\n\n\n> **W3: Missing detailed report on each metric's individual performance.**\n\nWe apologize for not including detailed sub-task performance tables due to limited space. We agree that this information is beneficial for interpreting model behaviors. Since the rebuttal is limited to 6,000 words, **we have moved the sub-task performance tables into the PDF posted under the global reply section of this rebuttal**. Please kindly refer to the tables there for more information.\n\n\n\n> **Q1: Incorporate human evaluations to support the credibility and relevance of the MR metric.**\n\nAs detailed in lines 207-209, we have reported the **92% of human-model agreement rates** on the error reason scoring. Below is the exact detail of our setup: \nWe randomly collected 100 data instances where the evaluating model correctly identified the solution correctness and the first error step over all the subjects. We then manually examine whether or not the proxy scoring model (e.g. GPT-4-Turbo-2024-04-09) has correctly scored the error reasons of the evaluating models. Below is the detailed composition of the ratio that the author agrees with the proxy scoring model:\n\n| Coding | Physics | Biology | Medicine | Chemistry | Logic | Math  |\n|--------|---------|---------|----------|-----------|-------|-------|\n| 7/8    | 12/13   | 21/21   | 12/12    | 15/17     | 15/16 | 10/13 |\n\nThe annotation time varies significantly across subjects, as some problems like coding and chemistry might take more than 10 minutes to evaluate, while subjects like biology will be easier to evaluate. We sincerely hope the author-model agreement rates and the agreement tables among the three different models above can relieve your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear PCs, SACs, ACs and reviewers: 

We sincerely appreciate your thoughtful review and insightful comments. We have tried our best to address your concerns one by one in the corresponding rebuttal sections. If our answers satisfy your queries, we would be grateful if you could consider revising your final rating to a higher score.

# PDF for Subtasks Performance Table

**Attached is the PDF of the breakdown performance table for models in all four metrics** (MR-Score, MCC, ACC_step and ACC_reason) that we can not enlist in the individual rebuttal section due to the character limit. This table should hopefully bring some insights about our design choice of MR-Scores and the subtask metrics:
1. **Metric Robustness**: Due to the progressive nature of the difinitions of our subtasks (e.g. the success of subsequent tasks depends on the previous ones), we can see the diminishing trend in the scores of MCC, ACC_step and ACC_reason. However, thanks to the design of our evaluation mechanism and metrics, **the score rankings of different models stay in relatively stable order across metrics.** In another words, we have not observed any model that excels in determining the solution correctness (thus high in MCC) but is unable to explain the rationale behind it (e.g. low in ACC_reason).

2. **Task Difficulties**: As shown in the breakdown table, the ACC_reason metric is more discriminative than the MCC metric for competent models but vice versa for the less competent ones. This aligns with our intuition that generally more difficult questions are more discriminative for strong candidates and the weaker ones are simply incapable of solving them whatsoever. **This phenomenon could in part explain why in general the MR-Score is not very sensitive to minor change in the weightings assigned to the subtasks**, since the differentiability of the subtask metrics tend to reconcile with each other under different scenarios. 

3. **Differentiability and Interpretability**: The weights of the MR-Score is ultimately decided by considering both the discriminative ability and the interpretability. To best differentiate models with different evaluation results, we conducted a thorough grid search to investigate the impact of the weightings. Since the weightings calcultated returned a few optimal instances, we deliberately selected the one that assign higher scores to more difficult tasks. **We believe the current weighting ratio strikes a good balance between interpretability and differentiation**: For example GPT-4-Turbo, Deepseek-v2-236B and Mistral-Large achieve 86.4% ,78.5% and 81.2% respectively in MMLU but score 43.2%, 29.4% and 21.3% in our benchmark.

Hope this table will help clarify some of your concerns regarding model performance and metrics design. 



Wishing you all the best,

Sincerely,

The Authors

Meta-review: This paper proposes Mr.Bean, a comprehensive benchmark that evaluates LLMs' capabilities of identifying incorrect reasoning steps rather than focusing on the accuracy of the final results. Compared to previous works sharing similar ideas, Mr.Bean's major contribution lies in that it significantly broadens the coverage and increases the scale. Experiments are comprehensive and reveal interesting limitations of many strong opensource models.

The response has addressed most of the reviewers' concerns. A major concern, as raised by SioJ and I share, is the lack of acknowledgement of previous works. The overall sentiment leans towards positive, and I recommend that the paper gets in. **The authors will want to revise their framing in the abstract and intro, to appropriately credit MR-GSM8K and MR-Math.**

At least one review is discounted due to quality and lack of participation in discussion.
Decision: Accept (poster)