Premise-Augmented Reasoning Chains Improve Error Identification in Math reasoning with LLMs
Authors: Sagnik Mukherjee
Abhinav Chinta
Takyoung Kim
Tarun Anoop Sharma
Dilek Hakkani Tur
Venue: ICML 2025 poster (2025)
Keywords: Mathematical Reasoning
Large Language Models
Primary area: deep_learning->large_language_models
TL;DR: This paper shows that in reasoning chains, LLMs can identify premises under which a particular step was written. Also using these premises improves error identification.

Abstract: Chain-of-Thought (CoT) prompting enhances mathematical reasoning in large language models (LLMs) by enabling detailed step-by-step solutions. However, due to the verbosity of LLMs, the resulting reasoning chains can be long, making it harder to verify the reasoning steps and trace issues resulting from dependencies between the steps that may be farther away in the sequence of steps. Importantly, mathematical reasoning allows each step to be derived from a small set of premises, which are a subset of the preceding steps in the reasoning chain. In this paper, we present a framework that identifies the premises for each step, to improve the evaluation of reasoning. We restructure conventional linear reasoning chains into Premise Augmented Reasoning Chains (PARC) by introducing premise links, resulting in a directed acyclic graph where the nodes are the steps and the edges are the premise links. Through experiments with a PARC-based dataset that we built, namely (Premises and ERrors identification in LLMs), we demonstrate that LLMs can reliably identify premises within complex reasoning chains. In particular, even open-source LLMs achieve 90% recall in premise identification.  We also show that PARC helps to identify errors in reasoning chains more reliably. The accuracy of error identification improves by 6% to 16% absolute when step-by-step verification is carried out in PARC under the premises.
Our findings highlight the utility of premise-centric representations in addressing complex problem-solving tasks and open new avenues for improving the reliability of LLM-based reasoning evaluations.

Premise-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
Sagnik Mukherjee !
Abhinav Chinta 11
Takyoung Kim
Tarun Sharma
Dilek Hakkani-Tin
Abstract
1. Introduction
Chain-of-Thouu (CoT) prompting enhances mathematical reasoning in large language models (LLMs) by enabling detailed step-by-stey solu- tions. However, due to the verbosity of LLMs, the resulting reasoning chains can be long, making it harder to verify the reasoning steps and trace issues resulting from dependencies between the steps that may be farther away in the sequence of steps. Importantly, mathematical reasoning allows each step to be derived from a small set of premises, which are a subset of the preceding steps in the reasoning chain. In this paper, we present a framework that identifies the premises for each step, to improve the evaluation of reason- ing. We restructure conventional linear reason- ing chains into Premise Augmented Reasoning Chains (PARC) by introducing premise links, resulting in a directed acyclic graph where the nodes are the steps and the edges are the premise links. Through experiments with a PARC-based dataset that we built, namely PERL (Premises and ERrors identification in LLMs), we demon- strate that LLMs reliably identify premises within complex reasoning chains. In particular, even open-source LLMs achieve 90% recall in premise identification. We also show that PARC helps to identify errors in reasoning chains more reliably. The accuracy of error identification im- proves by 6% 16% absolute when step-by- step verification is carried out in PARC under the premises. Our findings highlight the utility of tremise-centris representations in addressing complex problem-solving tasks and open new av- enues for improving the reliability of LLM-based reasoning evaluations.
shain-of-thought reasoning enhances problem solving by breaking down complex tasks into series of logical steps, improving the accuracy and clarity of decision-making pro- cesses. There has been a well-documented success of the reasoning capabilities of LLMs with chain of thought rea- soning (CoT; (Wei et al., 2023)) across applications such as embodied reasoning (Gao et al., 2023; Philipov et al., 2024), code generation (Jiang et al., 2024), and mathematical and scientific reasoning (Imani al., 2023; Ahn al., 2024). However, the corresponding evaluation methods and metrics have a significant limitation: they focus solely the correct- ness of the final answer, neglecting intermediate reasoning processes and rationales that contribute to it. Although the final answer serves as a proxy for the reasoning capability, it is not sufficient to judge the reasoning performance of the model. Hence, evaluating the final answer provides narrower view of the reasoning capabilities. Since the inter- mediate reasoning process is equally important as getting the correct final answer (Huang & Chang, 2023; Golovneva et al., 2023; Prasad et al., 2023), comprehensive evaluation of the reasoning chains is crucial to holistically understand- ing the reasoning capabilities of LLMs.
Evaluation of reasoning chains has been studied in litera- ture in the context of self-verification (Weng al., 2023). Previous work has shown that LLMs struggle to find rea- soning errors in chain-of thothhttt traces without the help of external verifiers (Stechly et al., 2024; Wu et al., 2024; Tyen et al., 2024), casting doubt on the overoptimism of LLMS' self-critique abilities. Existing research focusing on the evaluation of reasoning chains in LLMs can be broadly categorized into reference- based and reference-free meth- ods. Reference-based methods, which rely on the avail- ability of a ground truth reasoning chain (Welleck et al., 2021; Han et al., 2024; Tian et al., 2021), are reliable but are constrained by the significant cost of human annota- tions, restricting their application. In contrast, reference- free methods (Prasad et al., 2023; Golovneva al., 2023; Zhu et al., 2024) bypass the need for annotations, but suffer from two major drawbacks: (1) most such works assign only chain-level score rather than localizing specific er- rors, and (2) they often need fine-tuning of task-specific
'Equal contribution University of Illinois Urbana Champaign. Correspondence to: Sagnik Mukherjee <sag- nikm3 @ illinois.edu>..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Premisc-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
Question: given a- 10 and b- what are the mean of the following a+b, ab, a²b, ab*
Linear Reasoning Chain
Premise Augmented Reasoning Chain (PARC)
Step given w-bn-3
Step a-10 b- a1b-30
Step given 20-200 and 2-100, abb-10000
(b=200
Step 10000 12000) /4 5557.5
Question
1010,, 11b-10000
Step 2: 200
Step 4: civen a0-600 and b-20, 2b-2-12000
Stop 4: given 5b2502 and b-20, 40" 12200
$557.5
Correct step
Wrong Step
Accumulation Error
Figure 1. Comparison between Linear Reasoning Chain (IRRC) and our proposed PARC Tremise-Augmentt Reasoning Chain). The LRC (left), is linear and there no explicit premise link between steps. In PARC (right), premise links are explicitly established. enabling better identification of correct and incorrect steps. Accumulation errors can be traced back to faulty premises. Establishing these premises helps improve error detection with LLMs.
models, restricting their generalizability. One workaround could be using formal proof assistants like Lean (Moura & Ullrich, 2021), which natively support the verifiability of generated proofs (Yang et al., 2024; 2023; Murphy et al., 2024), but this requires a challenging formalitmaiinnionn of natural language text (Wu al., 2022) and often assumes the solution is already known for proof construction.
once the first mistake appears. Distinguishing accumulation errors from inherently flawed steps (mathematical errors or logical inconsistencies) is crucial for a holistic evalua- tion of CoT traces. Moreover, establishing explicit premise links between steps allows each step to be verified only un- der its premises which reduces irrelevant information for each verification step, thus improving the error detection for mathematical problem solving.
In our work, we focus on reference-free verification of LLM reasoning chains in the context of mathematical reasoning. Mathematical problem solving requires a series of deductive reasoning steps, where each step is performed under a small set of premises. We hypothesize that a step in a reasoning chain should be verified only under its premises. Previous work has shown that having unnecessary context hurts the performance of LLMs for solving math word problems (Shi et al., 2023). Thus, by excluding irrelevant context from the verifier, we make the verification of a reasoning step only conditioned on its premises, addressing the known susceptibility of LLMs to errors under distractors.
Our main contributions are as follows:
We demonstrate that off-the-shelf LLMs can detect premises for given step with high accuracy for mathe- matical reasoning, enabling the conversion of a Linear Reasoning Chain (LRC) to a Premise Premise-Augmented Rea- soning Chain (PARC).
We establish that verifying each step under its corre- sponding premises increases the accuracy of identify- ing errors and their types.
We restructure conventional linear reasoning chains (LRCs) into PARC Premise-Augmente Reasoning Chains), by identifying the premises of each step. Identifying those premises improves the traceability of a reasoning chain. We create corresponding dataset, called PERL (Premises and ERrors identification in Language models) to test LLMs' capability identifying premises as well as the effective- ness of premise augmentation in annotating both premises and errors. Additionally, refine the error taxonomy (as proposed in Golovneva et al. (2023)) for mathematical rea- soning by introducing new category: "accumulation error." This error arises when a reasoning step is correct in isola- tion but is derived from flawed premises (refer to step 5 in Fig. 1), resulting in error propagation throughout the chain; while such errors are common, existing work (Lightman al., 2023; Zheng et al., 2024; Daheim et al., 2024) has not addressed them beyond discarding all subsequent steps
We propose refined error taxonomy for mathemati- cal reasoning, introducing the notion of accumalation errors for steps that are locally correct but inherit up- stream errors.
We introduce and will release PERT,, a dataset of rea- soning chains annotated with premises and error types, to facilitate broader research premise- premise-centered rea- soning verification.
The rest of the paper is structured as follows: in Section we discuss relevant prior work in context of mathematical reasoning and verification of reasoning chains with LLMs. Section 3 introduces the research questions, alongside the necessary background as well as mathematical definition of premises, conversion of LRC to PARC, as well as how we identify errors with premises. Section 4 details PERL, our dataset to show effectiveness of the framework, and lastly,
2

---

Premisc-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
in section 5 we discuss the results and insights from our experiments.
premise extraction standalone step and refrains from making strong assumptions about how the reasoning chain is structured.
2. Related work
3. Method
Math Reasoning with LLMs. Mathematical and scien- tific reasoning tasks have become a primary testbed to assess the capabilities of large language models (LLMs) (Hendrycks et al., 2021a; Arora et al., 2023; Wang al., 2024). Solving these tasks requires complex planning, recall of relevant formulae, and grounding the solution to the given problem (Arora et al., 2023). A widely adopted approach to improve the reasoning of LLMs is to generate intermediate rationales, commonly known as chain-of-thought (CoT), before predicting final answer (Wei et al., 2023). Recent work explores alternative structures for organizing these solutions, including ofee-of-Thoughh (Yao et al., 2023) and Craphh-of-Thoughts (Besta et al., 2024).
In this work, we explore how the establishment of premise links improves the identification of eπrors in reasoning chains with LLMs. Our approach works in two steps. First, identify premises for each step and augment linear soning chain (LRC) to convert it into Premise Augmented Reasoning Chain (PARC). And then verify each step in the PARC under its premises only. Lastly, we do a graph traver- sal identify steps that are logically correct but have faulty premises identify accumulation errors. particular, we are interested in the following research questions.
RQ 1 word problem, can LLMs identify premises for each step?
Given sequential step-by-step answer to math
RQ 2 Given premise annotations, can LLMs identify errors in reasoning more faithfully?
Evaluation of Chain-of-Thoughtt. Although generating rationales can significantly improve model reasoning, sys- tematically evaluating these explanations remains chal- lenge. Traditional metrics focused word overlap em- bedding similarity (Celikyilmaz et al., 2021; Reiter, 2019) fail to capture logical soundness, especially for step-by-step deductions. To address this gap, methods such as ROSCOE (Golovneva et al., 2023), ReCEval (Prasad et al., 2023), and Socreval (He et al., 2024) offer reference-free evaluation frameworks that assess correctness and identify various cat- egories of errors. However, these methods generally provide single chain-level score and offer limited explainability, making it difficult to pinpoint specific error types in individ- ual reasoning steps. In contrast, our proposed framework provides a natural language characterization of the correct- ness of each step, providing ame-grained insights into the precise nature of errors.
RQ 3 Can LLMs perform the entire process end-to-end, i.e., given reasoning chain, can they identify premises for each step and detect errors?
3.1. Premise Augmented Reasoning Chains (PARC)
Let q represent the question, å the predicted answer, a the ground truth answer, and rst [51. $2....... the gener- ated reasoning chain composed of intermediate steps si, leading to â. Using CoT reasoning, the predicted answer à is derived by first generating reasoning chain rgt- The probability distribution for â can be expressed as:
PiM(â à q) = =iIM(ā rg)) X (IM( g),
where the reasoning chain rgg can be decomposed into in- termediate steps as:
Verifiers for Math Reasoning, Verifiers have proven ef- fective enhancing LLM-based reasoning. Uesato et al. (2022) and Lightman et al. (2023) demonstrate that pro- viding reward signals for intermediate steps can stabilize training and improve final performance. Deductive Beam Search (Zhu et al., 2024) similarly incorporates trained verifier to refine model outputs during inference, leading to higher accuracy for the final task. While these studies high- light the benefits of verifiers, a clear strategy for training a robust verifier, one that generalizes across diverse problem formats, remains elusive. The closest existing effort, Ling al. (2023), requires models encode premises in a partic- ular "natural program" style, which constrains applicability to general chains-of saains-of-thougee thought. In contrast, our method treats
[ PIM(S s;).
PIM(( I () = PIM((s I ④ x
Our objective is to identify errors in the intermediate steps S of the reasoning chain Tst-
The premises for a step s; are defined as the necessary and sufficient subset of prior steps, denoted as
P: S {syVj <}.
satisfying the following properties:
1. Verifiability: The correctness of s; is verifiable based on
I Our code and data is available htttp:://githui /Sagni kMukherjee, jee//rrjeePaaa PARC.
P: alone:
F(si P,)) 1.
3

---

Premisc-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
2. Minimality: The set Pi is minimal such that removing any element Sj Pi results in s; becoming unverifiable:
Algorithm 1 Constructing and Evaluating PARC
Input: [2........... sr]
// Step 1: Premise Extraction
F(x 1 Pi (s))) 00, Vsj PP:
(1)
for k 1 to do
Px F ($1....,.5-1))
Given premises P/ for step St, we convert linear reason- ing chain into PARC, where each step s; is augmented with its premises P, F is function that estimates whether step is verifiable or not.
end for
R' - (1).,.)).... Pr)]
// Step 2: Error Detection
for k = to do ISMATHENAATCALEERROOOO
8
gg [(s,, Pi), (s,,P)). ..... (s,, P.),,
k
BBLOGGCALLYIVOOOSSSTTTTTTT
& - E v end for // Step 3:
where l'g'' represents the transformed reasoning chain.
Accumulation Error Detection
3.2. Progressive Premise Mapping
for k to
if sg is correct then
To transform reasoning chain rg [s1, $2....... St] into reasoning chain (PARC), each step si must be explicitly linked to its premises P,. We explore two approaches to identify these premises: Aggregative Premise Mapping and Dyadic Premise Mapping.
for sj E Pk do
if S is incorrect then
Eg < ACCUMULATIONERROR break
end if end for
end if
3.2.1. AGGREGATIVE PREMISE MAPPING
end for R' c P1,),,. (s,,
In the Aggregative approach, the premises for each step S collectively identified by querying an LLM with the complete reasoning context up to the step. For a given question q. the reasoning chain so far reck and the next step sk. the LLM is prompted to output Pk.
from issues such as incorrect mathematical calculations (de- fined Mathematical Error), logical irregularities (defined as logical inconsistencies) and are evaluated independently of the broader reasoning context. Although significant, this focus on native errors tends to overlook another critical category of errors, called accumulation errors.
3.2.2. DYADIC PREMISE MAPPING
In the Dyadic approach, the task of identifying premises is reformulated pairwise evaluation. Instead of querying the LLM to identify all premises Px for Sk once, we evaluate whether each individual step (where i < k) serves as a valid premise for sg. For each pair (s,, sk), the LLM is queried to compute:
An accumulation error occurs when reasoning step valid in isolation but is built upon or more erroneous premises from earlier steps. Unlike native errors, accumulation errors emerge from the compounding effects of prior inaccura- cies, propagating through sequential steps. Formally, in the context of the reasoning chain, a step s; is classified as:
1, if sy is a valid premise for sk. I (sa s;)
otherwise.
1. Correct, if contains no logical or mathematical errors, and is performed under premises that are also correct
The premises for sk are then given by:
2. A native error if it contains inherent discrepancy (e.g.. miscalculation or logical inconsistency)
Px {s (sk s;) = 1, Vi k}.
3. An accumulation error if s; is logically valid, but at least one of its premises is incorrect.
3.3. Error Identification
Next, we present our setup for error identification and how premises play significant role in this task. We introduce taxonomy of error types that oceur in math reasoning with LLMs and then describe our approach to identify them.
By addressing both native and accumulation errors, we can achieve more comprehensive understanding of error dy- namies in sequential reasoning processes.
3.3.2. ERROR IDENTIFICATION
3. TYPES OF ERROR
Baseline. In the baseline approach, a large language model (LLM) is used in a zero-shot setting to classify the reasoning steps. For a given question q, the reasoning chain so far rek, and the next step proposed sk, the model
Traditionally, prior research has predominantly focused on identifying native errors, which refer to inaccuracies inher- ent to individual reasoning steps. These errors often arise

---

Premisc-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
Al constant temperature, the pressure of sample gas is inversely proportional to its volume. I have some hydrogen in a 3.67 liter container with pressure of 4 kPa. move
1.835 liter container at same temperature, what will new pressure be in kPa?
Solution
Baseline Verifier
Our Verifier
Step 1: We can use the formula for inverse proportionality between pressure (P) and volume (V) of gas
temperature, which given by: 1/V
Step 2: We rewrite this equation introducing constant of proportionality (k): P k/V
Step 3: Given that the initial pressure (P1) kPa and the initial volume (V1) 3.67 liters, we can write: P1 = k/ V1, 4 k 3.67
Stop 4: To find the value of can multiply both sides by 3.67: k = 3.67, k 14.68
Step 5: Now that we have the value of k, we can use it to find the new pressure (P2) when the volume (V2) is 1.835 liters: P2 k V2, P2 14.68 / 1.835
Mathematical
Step 6: find the new pressure, can divide 14,68 1.835: P2 7.98 kPa
Error
Step 7: Therefore, the new pressure of the hydrogen in the 1.835 liter container will be approximately 7.98 kPa.
Accumulation
Error
Figure 2. An examplc where the baseline method fails to detect errors, while verification method with established premise links successfully identifies the mathematical error in step 6, and the accumulation error in step 7.
queried to assign an error type Eg for according to the predefined error taxonomy (Correct Mathematical Error / Logical Error Accumulation Error): The model processes the entire context (q, rick, 8k) and generates a classification directly. The zero-shot prompt explicitly defines all error types guide the classification.
3. Accumulation Error: After identifying native errors, accumulation errors are identified by analyzing the depen- dency graph of the reasoning chain. The premises Px form directed edges in the graph, where each step depends on its premises. A Depth-First Search (DFS) traversal is per- formed, and step Sı is classified an Accumulation Error if it satisfies the following criteria: 1. Sg itself is classified as correct, and 2. At least one premise sy e Pk (where k) is classified as incorrect.
Proposed Approach. Our proposed approach refines the classification process by situating each reasoning step explicitly within the context of its premises Pt. Algorithm illustrates the overall pipeline of conversion of LRC to PARC and step-by-step error classification. The prompts for the following reported in Appendix 16 and 17.
4. PERL: Premises and ERrors identification in Language models
In order to test the ability of LLMs to identify premises and error categories, we designed a testbed. We used existing datasets of math word problems to create PERL, our testhed for step-level premise and error annotations.
To detect mathematical errors, we prompt an LLM to assess whether contains a mathe- matical error by analyzing the step in isolation, excluding the broader reasoning chain.
1. Mathematical Error:
Generating Reasoning Chains: In order to generate the reasoning chains, we used two popular benchmarks (i) GSMSK (Cobbe al.. 2021), collection of 8,500 grade school math word problems, and (ii) MATH (Hendrycks et al., 2021b), dataset of 12,500 challenging competi- tion level math word problems. In addition, we use the (iii) Orca-Math dataset by Mitra et al. (2024), a synthetic dataset of 200K math problems alongside solutions written by GPT4Turbo, and the (iv) MetaMathQA dataset by (Yu et al., 2023). We first randomly sampled 1000 examples
Logical Inconsistency: For the detection of logical inconsistencies, restrict the evaluation to the premises Pk of the reasoning step s&. The LLM is prompted to determine whether 5k is logically consistent with its premises. This ensures that the step's 's validity evaluated in the context of the dependencies defined by Px. without considering unrelated parts of the reasoning chain.
5

---

Premisc-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
GSM8K
MATH
Orca Math
MetaMathQA
Model Name
Precision
Recall
F1
Precision
Rocall
F1
Precision
Recall
F1
Precision
Recall
F1
Llama 3.1 8b
65.64
89.33
75.66
$1.00
82.40
62,94
54.65
79.77
64.85
$3.92
81.42
64.75
Llama 3.1 70b
81.26
97.35
88.65
69.80
96.82
81.11
71.05
96.73
81.91
73.10
96.46
83,05
Qwen 2.5 7b
67.42
63,53
65.36
55.87
59.73
57.70
63.08
72.64
67.05
55.54
61.66
58,14
Qwen 2.5 32b
84.07
95.35
89.34
72.56
88.55
79.74
74.27
90.63
81.62
75.57
90.89
82.49
GPT4o-mini
72.54
86.18
78.75 75
57.72
69.29
62.96
60.12
73.96
66.31
60.77
76.70
67.71
GPT-4o
85.93
94.42
89.96
69.40
89.78
78.28
71.12
92.17
80.79
75.49
90.37
81.85
Table 1. Precision. Recall and Fl scores for premise identification under the Aggregative approach. Note that ideally this needs to be high recall system, because missing even single premise step could hurt the verifiability of the reasoning chain.
Positives
Negatives
Synthetic Negatives
Model Name
Precision
Recall
F1
Precision
Recall
F1
Precision
Recall
Fl
GPT4o-mini Aggregative
74.93
84.81
79.56
70.85
87.01
78.10
71.84
86.73
78.59
Dyadic
75.04
82.82
78.74
64.57
71.12
67.68
55.18
53.98
54.57
GPT4o
Aggregative 89.38
93.66
91.47
84.97
94.95
89.69
83.43
94.65
88.72
Dyadic
75.32
99.01
85.55
67.70
92.68
78.25
67.27
79.42
72.84
Table Precision, Recall and Fl scores for the premise mapping task for the GSM8K dataset under the aggregative and dyadic approaches. We saw consistent drop in Fl score for the dyadic approach, in spite of it being computationally more expensive.
from the GSM8K and MATH test split and the Orca-Math and MetaMathQA training split (since these are training datasets). We began by using the open weight Large Lan- guage Model Llama-3. Instruct (Grattafiori et al., 2024) to generate step-by-step reasoning chains. Next, we catego- rized the reasoning chains as positive or negative, depending on whether the final answer matches the ground-truth an- swer. Next, we randomly sampled 50 positive (correct) and 50 negative (incorrect) reasoning chains. To expand our dataset, we employed GPT-4o systematically intro- duce mathematical or logical errors into the correct reason- ing chains, creating additional synthetic negative examples (since prior work also focused on such synthetic negatives). When introducing errors, we ensured that the subsequent steps were appropriately modified reflect the impact of the initial error. In contrast with existing works such Golovneva et al. (2023), where the perturbation in step is not followed up in subsequent steps, our synthetic negatives are more realistic.
only 4 out of 71 had a discrepancy and 5 out of 71 had issues in error annotations. For premise identification, the discrepancy typically indicates additional steps as premises. However, for error annotation, the annotation errors do not have a clear pattern and manifest themselves in forms of misclassification of the correctness of the step. The prompts used to generate the annotations of the premises and errors are in Appendix A.3, A detailed summary of the dataset statisties can be found in Appendix A.2
5. Results and Discussion
For detailed information on our experimental setup, includ- ing model configurations and implementation details, refer to Appendix A.
5.1. Premise Mapping
In this section we try to answer RQI - "Can LLMs identify premises to convert a Linear Reasoning Chain to a PARC?"
Premise and Error Annotation: Next, for each step in the reasoning chain, we identified the ground truth premises for each step using the OpenAl ol Prevveww model (OpenAl et al., 2024). Then, we use the same ol ol-Preview model to map each step to the predefined taxonomy of errors or mark it as correct, to create grouhd-truth error annotations. Upon obtaining the annotated data sets, 2 authors manually went through 10% of the generated data points to identify the discrepancy between the human annotations and the annotations done by the ol model. Manual inspection of 71 samples showed that, the case of premise annotations,
Metrics: For identification of premises, we use precision, recall, and score as metrics. However, we would want to highlight to the reader that the most important metric here is Recall. Since marking an extra step as a premise will not hurt the verifiability of the step, missing one will hurt its verifiability and all subsequent steps that rely on this step as premise. Further, we note that some reasoning chains have a higher number of reasoning steps compared to others. In order to ensure robustness of the metrics, we first compute them the level of each data point and later average the
6

---

Premisc-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
metrics across the dataset. We tried two approaches for the task of premise mapping.
quite thin, and models often classify one as the other, so we merge these two error categories into a single error type Error, while keeping Accumulation Error separate. Further- more, since an early erroneous step for a solution containing multiple steps could cause the number of errors to skew toward accumulation errors, we first normalize the accuracy by the number of steps in a datapoint, and later normalize across the dataset.
A Aggregative Approach: Table contains the results of the Aggregative approach for a suite of models and datasets. From the table, it is evident that LLMs are efficient in iden- tifying premises for a given step. In particular, Llama 3.1 70b (Grattafiori et al., 2024), Qwen 2.5 32b (Qwen et al., 2025) and GPT4o are very efficient in this task, all having a recall greater than 90%. It also shows that these models can convert an LRC to a PARC with high accuracy.
Results: Table 3 presents the experimental results for the error identification task. We evaluate performance under two setups: under complete context input (Full Context), on the other hand, the Model Premises setup denotes the end-to-end approach, wherein the model is tasked with first identifying the relevant premises and subsequently utilizing them. All models are provided with the question and the solutions generated up to the current step and are tasked with predicting the error type observed in the current step.
We note that the scale of the model is an important factor. The increase in recall varies from 8.2% to 32.2% when we scale up for the same model class, and the pattern is consistent. Tables 9, 10, 12 and 11 in the appendix contain the detailed metrics across the splits Positive, Negative and Synthetic negatives for GSM8K, MATH, OrcaMath and MetaMathQA respectively.
Dyadic Approach: Table 2 provides a comparative analy- sis of precision, recall, and Fl scores for the GSM8K dataset using two models for the aggregative and dyadic approaches. This approach causes the LLM to be more biased to mark step as a premise, causing precision to drop significantly. In addition, this approach has time complexity of Onn), where n is the number of reasoning steps. Note that for almost all cases the precision drops, and the change in recall gives a mixed view on how this approach might be helpful.
Premises Help Error Identification It is evident from Table 3, that providing only the premise steps as context (in- stead of providing the full context) improves the accuracy of error detection with LLMs. Figure 2 shows one such exam- ple, where the baseline approach (using an L.C,, and error detection performed in the full context) could not identify errors, but our approach was able to successfully identify the mathematical and accumulation errors. Our results show that the models are not as biased towards marking step as correct, when given a more precise context (no distractor). Hence, upon providing only the premises, identification of errors becomes much more robust. However, is important to note that this also comes at the cost of a drop of accuracy for correct steps, this can be observed from comparing the Pos column for Full context and Model Premises. However, this accuracy drop is much smaller for larger models. In addition, our approach impacts larger and more capable models more than smaller models.
Our analysis revealed that this methodology often led the model to incorrectly classify second- or higher-order de- pendencies as premises, even when the current step did not directly rely on them. This misclassification contributed to the observed degradation in precision.
5.2. Error Identification
In this section, we address RQ2 and RQ3, i.e. is error identification more robust when we only use the premises for a given step context. We explain the experimental setup for error identification for the baseline, as well as our approach. in the previous sections. For the identification of errors under premises, we consider two settings. For the first one, we use the oracle premise annotation (from the ground truth). For the second, we first identify the premises with the LLMs and then use those premises for the identification of errors. Note that in the previous section we established that LLMS can identify premises with a high recall, and in this experiment, we exploit that. The prompts we used for our experiments are shared in Appendix A.5.
Models Struggle With Error Detection, Especially Accu- mulation Errors: We find that, while capable of identifying correct steps, all models struggle with identifying errors. This can be attributed to the fact that models are inherently biased toward marking a step as correct. This issue was also pointed out by Ling et al. (2023). Identifying errors with directly connected premises as context yields significantly better performance than simply providing the full context. Moreover, models struggle to identify accumulation errors. Table 5 contains the accuracies across error types in the neg- ative split of GSM8K. Note that, in full context, accuracy is quite high for correct steps, but lower for erroneous steps (mathematical and logical errors), and significantly lower for accumulation errors. However, in our approach, the identification of errors becomes significantly robust, with a minor drop in performance in identifying correct steps.
Metrics: For this task, we report the average accuracy for the identification of error types for each step. In our analysis, we observe that the boundaries between the error types Mathematical Error and Logical Inconsistency are
7

---

Premise-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
GSM8K
MATH
Orca Math
MetaMathQA
Model
Nog
Syn
Pos
Avg
Neg
Syn
Pos
Avg
Neg
Syn
Pos
Avg
Neg
Syn
Pos
Avg
Llama 3.1 8b
Full Context
48.87
$7.68
91.9
58.6
46.91
60.29
90.61
60.2
43.26
51.2
96.7
55.06
50.47
55.15
95.17
59.45
Model Premises
54.61
76.74
65.44
64.53
52.16
59.83
50.96
54.88
55.57
59.82
67.36
59.22
54.52
69.13
62.63
61.78
Llama 3.1 70b
Full Context
58.35
77.03
98.61
71.37
55.55
72.83
96.24
69.77
50.33
64.25
96.74
63.52
55.59
73.13
97.93
69.49
Model Premises
74.51
86.22
94.69
81.92
70.94
79.29
77.91
75.45
64,95
72.28
93.37
72.52
65.69
82.74
89.71
76.52
Qwen 2.5 Full Context Model Premises
40.88
54.13
100
54.94
40.47
53.60
98.07
56.26
41.20
53.69
100
55.85
43.36
53.18
99.39
56.26
53.03
78.40
90.37
69.73
58.47
66.87
82.72
65.96
52.07
71.12
84.28
65.38
54.98
73.87
91.02
68.43
Owen 2.5 32b
Full Context Model Premises
46.76
69.26
99.8
63.56
49.58
66.5
98
65.11
49.44
63.01
99.5
63.06
49.7
67.8
99.6
65.02
65.58
86.34
96.95
79.37
68.61
76.96
88.71
75.57
61.89
69.24
95.25
70.26
62.18
85.60
97.32
77.40
Qwen 2.5 72b
Full Cuntext
47.48
67.17
100
63.11
53.19
67.87
99.79
67.52
50.41
60.33
100
62.50
47.31
63,59
99.05
62.20
Model Premises
69.87
85.
96.38
80.56
69.51
79.92
90.84
77.28
69.45
71.34
94.19
74.41
64.73
85.52
98.12
78.53
4PT44-mini
Full Context
52.68
73. 14
95.02
66.68
68
50.41
62.39
93.4
63.03
55.22
61.18
98.6
64.59
51.52
65.42
97.35
64.47
Model Premises
58.25
80.81
86.72
72.33
65.84
72.4
71.88
69.45
61.18
72.48
85.94
69.93
57.04
81.63
91.15
72.51
GPT-40
Full Context
53.81
75.3
98.33
68.52
50,9
68.41
98.57
66.52
59.55
65.75
100
68.55
56.26
71.72
99.3
69.41
Model Premises
65.23
86.67
99.76
79.82
70.12
76.67
91.84
76.45
66.33
71.74
93.14
72.96
67.28
88.19
89.85
79.41
Table Accuracy Error identification of various models under Full Context (baseline) and Premises settings across different datasets. Note that Neg, Syn and Pos means Negative, Synthetic Negatives and Positive splits of PERL, as explained carlier. For each dataset, we highlighted the models that benefit the most from the PARC.
Model
GSMSK MATH Orca-Math
MetaMath
Model
Correct
Error
Acc. Error
Avg
Llama 3,1 70b
Llama 3.1 70b
Oracle Premises
81.46
74.68
73.45 72.52
76.05
Full Context
96.79
60.87
12
58.35
Model Premises
81.92
75.45
76.52
Oracle Premises
90.18
75.25
55.63
74.41
Model Premises
88.78
75.25
57.54
74.51
Qwen 2.5 72b
Oracle Premises
80.58
79.50
74.87
79.49
GPT-4o
Model Premises
80.56
77.28
74.41
78.53
Full Context
95.75
48.2
13.2
53.81
Oracle Premises
96.83
60.02
41.79
67.03
GPT-40
Model Premises
94.15
55.
44.91
65.23
Orucle Premises
78.74
76.79
75.56
82.88
Model Premises
79.82
76.45
72.96
79.41
Table 5. Error identification accuracy for cach type of steps in ground truth. Acc. Error means Accumulation Errors
Table Comparison of Error identification under oracle premises vs model generated premises. Since these models have high recall premise identification, we observe that the error identification accuracy is comparable.
the accuracy of error identification improves when oracle premises are provided. However, it is noteworthy that for most models, the performance remains comparable to that achieved with oracle premises. This can be attributed to the fact that all these models can identify premises with a recall of higher than 90.
The average performance of all models synthetically generated negatives is consis- tently higher compared to the performance true negatives (evident from comparing the Neg and Syn columns in Table 3). This observation highlights an important insight: per- turbing reasoning chains does not serve as a reliable proxy for evaluating model's ability to identify genuine errors in reasoning. While prior work has focused on such synthetic negatives, it might provide overoptimistic view of LLMs' ability to detect errors.
Synthetic Negatives are Easier:
5.3. Experiments on ProcessBench
Beyond the PERL dataset, we also evaluate our framework on ProcessBench (Zheng al., 2024), a benchmark that contains auman-annotated stepwise labels for mathemati- cal solutions. Unlike PERL, ProcessBench only includes annotations up to the first incorrect step in each solution.
Oracle Premises vs Model Generated Premises: In Table 4, we present results of an ablation, where instead of provid- ing the model generated premises, we provide the ground truth premises (oracle premises from PERL). As anticipated,
Experimental Setup. We focus on the step-level error identification task, following the protocol used by Zheng et al. (2024). Specifically, for correct solutions (i.e., ones
8

---

Premisc-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
Model
Method
Correct
Wrong
F1
Delta
iments with our annotated dataset, PERL, we empirically show that error identification under premises in PARC is more reliable and has higher accuracy than error identifica- tion under full context in LRC. We also show that LLMs can convert an LRC to PARC with no additional guidance, and then can do error identification under the premises in PARC. Further, we show that accumulation errors are partic- ularly challenging to detect, and our method improves their identification by verifying each step under its premises.
Qwen 2,5 7B
Baseline Ours
66.3
36.7
47.2
60.1
38.6
47.0
-0.2
Qwen 2.5 32B
Baseline Ours
97.9
43.0
59.8
95.9
55.1
70.0
+10.2
Qwen 2.5 72B
Baseline Ours
98.4
61.4
75.6
97.8
59.7
74.1
-1.5
Llama 3.1 8B
Baseline Ours
17.1
36.7
23.3
33.7
37.8
35.6
+12.3
Llama 3.1 70B
Baseline Ours
77.7
57.5
66.1
89.6
70.0
78.6
+12.5
A Acknowledgments
Table Performance comparison on the GSM8K split of Process- Bench with Baseline vs. PARC (Ours implies premise augmented verification) for various model.
This research project has benefited from the Microsoft Ac- celerate Foundation Models Research (AFMR) grant pro- gram through which leading foundation models hosted by Microsoft Azure along with access to Azure credits were provided to conduct the research.
Model
Method
Correct
Wrong
F1
Delta
Qwen 7B
Baseline Ours
46.0
25,4
32.7
45.6
41.2
43.3
+10.6
Qwen 2.5 32B
Baseline
90.0
22.4
35,9
Ours
86.9
53.9
66.5
+30.7
Impact Statement
Qwen 2.5 72B
Baseline Ours
88.5
33.7
48.8
The broader impact of this research extends to multiple do- mains where step-by-step reasoning is critical, including automated theorem proving. educational AI tutors, and sci- entific discovery. By enabling more reliable seff-verfficttion mechanisms in LLMs, our approach contributes to the long- term goal of enhancing the transparency and trustworthiness of Al systems in high-stakes applications.
86.7
53.9
66.5
+17.7
Llama 3.1 8B
Baseline Ours
5.6
19.1
8.7
11.0
27.5
15.7
+7.1
Llama 3.1 70B
Baseline Ours
32.4
32.8
32.6
61.6
55.4
58.3
+25.7
Table 7. Performance comparison on the MATH split of Process- Bench with Baseline PARC (Ours implies premise augmented verification) for various model.
Potential ethical considerations include the risk of overre- liance on LLM-generated verification without human over- sight, particularly in domains where reasoning errors can have significant real-world consequences.
where the final answer is correct), every step is marked as correct. For incorrect solutions, the task is to identify the first incorrect step in the chain; all subsequent steps are automatically deemed irrelevant by design of the benchmark. We adopt the same prompting strategy as in our previous experiments, with LLMs classifying each step as correct or incorrect. The baseline is a standard critic model as proposed by (Zheng et al., 2024) that identifies the first erroneous step from the solution as a whole.
This paper primarily aims to advance the field of Machine Learning by proposing a structured approach to reasoning evaluation. Although we acknowledge that any advances in LLM-based reasoning may have broader societal impli- cations, we do not foresee any immediate ethical concerns beyond those generally associated with the development of Al reasoning systems.
Results and Observations. Tables 6 and 7 Our findings on ProcessBench (on the GSM8K and MATH splits) align with the trends we observe on PERI.. Premise-augmented verification continues to outperform verification conducted in the full reasoning chain. We see consistent gains from evaluating each step under its extracted premises and classi- fying its correctness accordingly. This further strenghtens our previous findings.
References
Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., and Yin, W. Large language models for mathematical reasoning: Progresses and challenges, 2024. URL https: xiv. org/abs//////ooooooooooooo... 2402 00157.
Arora, D.,, Singh, H., and Mausam. Have LLMs advanced enough? a challenging problem solving benchmark for large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7527-7543, Singapore, December 2023. Association for Computational Linguistics. doi: 10.
6. Conclusion
In this paper, we introduced a framework to enhance the evaluation of mathematical reasoning chains in large lan- guage models by transforming Linear Reasoning Chains into Premise Augmented Reasoning Chains. Through exper-

---

Premisc-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
1p-main.468. URL https: http:ps:///aaaaahlloooooooo org /2023. eenml-mmii..46..
C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong,
C., Ferrer, C. C., Nikolaidis, C.,, Allonsius, D., Song, D., Pintz, D., Livshits, D., Wyatt, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia- Garcia-Olano, D., Perino, D., Hupkes,
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Pod- stawski, M., Gianinazzi, L., Gajda, J.. Lehmann, T., Niewiadomski, H., Nyczyk, P., and Hoefler, T. Graph of thoughts: Solving elaborate problems with large lan- guage models. Proceedings of the AAAI Conference on Artificial Intelligence,
D., Lakomkin, E., AlBadawy, E., Lobanova, E.,, Dinan, E., Smith, E. M., Radenovic, F., Guzmán, F., Zhang, F,, Synnaeve, G., Lee, G., Anderson, G. L., Thattai, G., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Ko- revaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I., Misra, I., Evtimov, I., Zhang. J., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock. J., Hong. J. Lee, J., Fu, J., Chi, J., Huang, J., Liu, J.,, Wang. J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J.. Alwala, K. V., Prasad, K., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, Keatala-Yeary, Yeary, I.,, van der Maaten, I.., Chen, I.., Tan, L.,, Jenkins, L.,, Martin, L., Madaan, L., Malo, L., Blecher, L., Landzaat, L., de Oliveira, L., Muzzi, M., Pasupuleti, M., Singh, M., Paluri, M., Kardas, M., Tsimpoukelli, M. Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bushlykov, N., Bogoychev, N., Chatterji. N., Zhang, N., Duchenne, O., Çelebi, O., Airassy, P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P., He, Q. Dong, Q. Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Maheswari, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabasappa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie. S., Narang, S., Raparthy, S., Shen, S., Wan. S., Bhosale, S., Zhang, S., Vandenhende, S.,, Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speck- bacher, T., Mihaylov. T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V. Vogeti, V., Albiero, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Wang, X., Tan, X. E., Xia, X., Xie, X., Jia, X., Wang, X., Gold- schlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z., Chen, Z., Papakipos, Z., Singh, A., Srivastava, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Teo, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poul- ton, A., Ryan, A., Ramchandani, A., Dong, A., Franco, A., Goyal, A., Saraf, A., Chowdhury, A., Gabricl, A.. Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B.. Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B.,
March 2024. URL
ISSN 2159-5399. doi: 10. 10..1609/aaaiii.333116666 http:/ /tpp://dxxxxx /xx doi www.doo..ooooooooooooooooo.o.o / w0w//aai.....//////.///aaaa..//a v38116 .29720.
Celikyilmaz, A., Clark, E., and Gao, J. Evaluation of text generation: A survey, 2021. URL https://aaxii org/abs/200 2006. 6 14799.
Cobbe, K. Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers solve math word problems, 2021. URL https:// org/abs http/abb/2211.... 2110. 14168.
K., Chiu, K., Bhalla, K., Lakhotia, K., Rantala-
Daheim, N., Macina, J., Kapur, M., Gurevych, I., and Sachan, M. Stepwise verification and remediation of student reasoning emrors with large language model tu- tors, 2024. URL htttp::///aaat org/ abs/2407 .99136.
Gao, Q. Thattai, G., Shakiah, S., Gao, X., Pansare, S., Sharma, V., Sukhatme, G., Shi, H., Yang, B., Zhang, D., Hu, L., Arumugam, K., Hu, S., Wen, M., Guthy, D., Chung, S., Khanna, R., Ipek, O., Ball, L., Bland, K., Rocker, H., Johnston, M., Ghanadan, R., Hakkani-Tur, D., and Natarajan, P. Alexa arena: A user-centric inter- active platform for embodied ai. In Oh, A., Naumann, T., Globerson, A., Saenko. K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Sys- tems, volume 36, pp. 19170-19194. Curran Associates, URL s. s. cc/ httc:/pper__iilesepppee////////p 758f0b95e19abce 09 Ratasets_and_Benchnnee
Golovneva, O., Chen, M., Poff, S., Corredor, M., Zettle- moyer, L., ZarandZarandi, M., and Celikyilmaz, A. Roscoe: suite of metrics for scoring step-by-step rea- soning, 2023. URL /ttpss://arxiv. org/ abs/22 17917919.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang. A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra,
10

---

Premisc-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
Montalvo, B., Parker, C., Burton, C., Mejia, C., Liu, C.,, Wang. C.,, Kim, C., Zhou, C., Hu, C., Chu, C.-H.,, Cai, C., Tindal, C., Feichtenhofer, C., Gao, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E., Jamil, E., Mont- gomery, E., Presani, E., Hahn, E., Wood, E., Le, E.-T., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, E. Kreuk, F., Tian, F., Kokkinos, F., Ozgenel, F., Cag- gioni, F., Kanayet, F., Seide, F., Florez, G. M. Schwarz, G. Badeer, G., Swee, G., Halpern, G., Herman, G., Sizov, G. Guangyi, Zhang, Lakshminarayanan G., Inan, H., Shojanazeri. H., Zou, H., Wang. H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Zhan, H. Damlaj. I., Molybog, I., Tufanov, I., Leontiadis, I., Veliche, I.-E., Gat, I., Weissman, J.. Geboski, J., Kohli, J.. Lam, J., Asher, J., Gaya, J.,B., Marcus. J., Tang, J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang. J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg. J., Wang. J., Wu, K., U. K. H., Saxena, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K. Michelena, K., Li, K., Jagadeesh, K. Huang, K., Chawla, K., Huang. K., Chen, L.., Garg. L., A, Silvv,, L., Bell, L., Zhang, L., Guo, L., Yu. L., Moshkovich, L.., Wehrstedt, I.., Khabsa, M., Avalani, M., Bhatt, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Liu, M., Seltzer, M. L., Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M.. Wang, M.. Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Mehta, N., Laptev. N. P.. Dong, N., Cheng, N., Chernoguz. O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj, P., Liang, Q. Alao, R., Rodriguez, R., Ayub. R., Murthy, R., Nayani, R., Mitra, R., Parthasarathy, R., Li, R., Hogan, R., Battey, R., Wang, R., Howes, R., Rinott, R., Mehta, S., Siby, S., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Mahajan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Patil, S., Shankar, S., Zhang. S., Zhang, S., Wang. S., Agarwal,, S.. Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satter- field, S., Govindaprasad, S., Gupta, S., Deng, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Koehler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V.. Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Ionescu, V., Poenaru, V., Mi- hailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wu, X., Wang, X.,
Qi Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Zhao, Y., Hao, Y., Qian, Y., Li, Y., He, Y., Rait, Z. DeVito, Z., Rosnbrick, Z.,, Wen, Z., Yang, Z., Zhao, Z. and Ma, Z. The llama 3 herd of models, 2024. URL https: /apps://axxxa
2407. 21783.
Han, S., Schoelkopf, H., Zhao, Y., Qi. Z., Riddell, M., Zhou, W., Coady, J., Peng. D., Qiao, Y., Benson, L.., Sun, L., Wardle-Solano, A., Szabo, H., Zubova, E., Burtell, M., Fan, J., Liu, Y., Wong, B., Sailor, M., Ni, A., Nan, L.,, Kasai, J. Yu, T., Zhang, R., Fabbri, A. R., Kryscinski, W., Yavuz, S., Liu, Y., Lin, X. V., Joty, S., Zhou, Y., Xiong, C., Ying. R., Cohan, A., and Radev, D. Folio: Natural language reasoning with first-order logic, 2024. URL https://azae wrg.iiioorg//aeee hwtgg//bs///ee /ww.ab///2000 02000000..000.....
He, H., Zhang, H., and Roth, D. Socreval: Large language models with the socratic method for reference-free rea- soning evaluation, 2024. URL https: /tatps:://rxiv. arxiv. or g //abs/230 2310. 00074.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding, 2021a. URL https: //arxiv. org/ http//bb//00000 03000
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. NeurlPS, 2021
Huang. J. and Chang, K. C.-C. Towards reasoning in large language models: A survey, 2023, URL https: /ttts://aau /ar xiv. org/ abs 221222212221 10403.
Imani, S., Du, L., and Shrivastava, H. MathPrompter: Mathematical reasoning using large language models. In Sitaram, S., Beigman Klebanov, B., and Williams, J. D. (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pp. 37-42, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 1118665//112222221lllooooe
URL htttttt/ss//cc //ac lanthology hwwphhoooooyyygg//2220 org/ acl -cl---a--0000 innuuiirrtr..r.. 4/.
Jiang, J., Wang, F., Shen, J., Kim, S., and Kim, S. A survey on large language models for code generation, 2024. URL https: / wwwpawwwwwaaaaaaaaaaa.e.. abs 240/ 00515.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez. J. E., Zhang, H., and Stoica, Efficient memory management for large language model serving with pagedattention, 2023. URL https::: /ttp:///axvvvvvviiv g/ abs/b/// /209 66180.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and
Wu, X., Gao, X., Kleinman, Y., Chen, Y., Hu, Y., Jia, Y.,

---

Premisc-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
Cobbe, K. Let's verify step by step, 2023. URL https: //arxiv. abs /2305 20050.
Ahmad, L.., Yang, L., Liu, L., Maksin, L., Ho, L., Fedus, L., Weng, I.., Li, L.,, McCallum, L.., Held, L.,, Kuhn, L., Kondraciuk, L., Kaiser, L., Metz, L.,, Boyd, M., Trebacz, M., Joglekar, M.. Chen, M., Tintor, M., Meyer, M., Jones, M., Kaufer, M., Schwarzer, M., Shah, M., Yathaz, M., Guan, M. Y., Xu, M., Yan, M., Glaese, M., Chen, M., Lampe, M., Malek, M., Wang, M., Fradin, M., McClay, M., Pavlov, M.. Wang. M., Wang. M., Murati, M.. Bavar- ian, M., Rohaninejad, M., McAleese, N., Chowdhury, N., Chowdhury, N., Ryder, N., Tezak, N., Brown, N., Nachum, O., Boiko, O., Murk, O., Watkins, O., Chao, P., Ashbourne, P., Izmailov, P., Zhokhov, P., Dias, R., Arora, R., Lin, R., Lopes, R. G., Gaon, R., Miyara, R., Leike, R., Hwang, R., Garg, R., Brown, R., James, R., Shu, R., Cheu, R., Greene, R., Jain, S., Altman, S., Toizer, S.,. Toyer, S., Miserendino, S., Agarwal, S., Hernandez, S., Baker, S., McKinney, S., Yan, S., Zhao, S., Hu, S., Santurkar, S., Chaudhuri, S. R., Zhang, S., Fu, S., Papay, S.,, Lin, S., Bal- aji. S., Sanjeev, S., Sidor, S., Broda, T., Clark, A., Wang, T., Gordon, T., Sanders, T., Patwardhan, T., Sottiaux, T., Degry, T., Dimson, T., Zheng, T., Garipov, T., Stasi, T., Bansal, T. Creech, T., Peterson, T., Eloundou, T., Qi, V., Kosaraju, V., Monaco, V., Pong, V., Fomenko, V., Zheng, W., Zhou, W., McCabe, W., Zaremba, W., Dubois, Y., Lu, Y., Chen, Y., Cha, Y., Bai, Y., He, Y., Zhang, Y., Wang, Y., Shao, Z., and Li, Z. Openai ol system card, 2024. URL https: a arxiv. orr.oog////aooo /tsp//////222// /212..1..67220..
Ling, Z., Fang, Y., Li, x., Huang, Z., Lee, M., Memisevic, R., and Su, H. Deductive verification of chain-of-thought reasoning, 2023. URL pesttps//aaxxe org/ abs/ 2306. 03872.
Mitra, A., Khanpour, H.,, Rosset, C.,, and Awadallah, A. Orca-math: Unlocking the potential of sims in grade school math, 2024. hRtp::ts:///aaaa
s/2402. 117302.11880. 4830.
Moura, L. d. and Ullrich, S. programming language. In Platzer, A. and Suteliffe, G. (eds.), Automated Deduction CADE 28, pp. 625-635, Cham, 2021. Springer International Publishing. ISBN 978-3-030-79876-5.
The lean theorem prover and
Murphy, L.., Yang, K., Sun, L,,
Li, Z.., Anandkumar, A., and Si, Autoformalizing euclidean geometry, 2024. URL https: wttp:/iiioor//oooo awtpgg//////// 22055 17216.
OpenAl. Openai ol system card. Technical report, Decem- oo- ber 2024. URL. hwwps:ttp/s.cc cdn. openai i com/on com/ol-
system--ard-20241200. pdf.
OpenAI, Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beu- tel, A., Carney, A., Iftimie, A., Karpenko, A., Passos, A. T., Neitz, A., Prokofiev, A., Wei, A., Tam, A., Bennett, A., Kumar, A., Saraiva, A., Vallone, A., Duberstein, A., Kondrich, A., Mishchenko, A., Applebaum, A., Jiang, A., Nair, A., Zoph, B., Ghorbani, B., Rossen, B., Sokolowsky, B., Barak, B., McGrew, B., Minaiev, B., Hao, B., Baker, B., Houghton, B., McKinzie, B., Eastman, B., Lugaresi, C., Bassin, C., Hudson, C., Li. C. de Bourcy, C., Voss,
Philipov, D., Dongre, V., Tur, G., and Hakkani- D. Sim- ulating user agents for embodied conversationallatt 2024. URL https: arxiv .o.. /tabb///41100 23352
Prasad, A., Saha, S., Zhou, X., and Bansal, M. Receval: Evaluating reasoning chains via correctness and informa- tiveness, 2023. URL https: /ttp:://axxvvvvvv abs/ 2304.10703.
C., Shen, C., Zhang, C., Koch, C., Orsinger, C., Hesse, C., Fischer, C., Chan, C., Roberts, D., Kappler, D., Levy, D. Selsam, D.. Dohan, D., Farhi, D., Mely, D., Robinson, D.. Tsipras, D., LI, D., Oprica, D., Freeman, E., Zhang, E., Wong, E., Proehl, E., Cheung, E., Mitchell, E., Wal- lace, E., Ritter, E., Mays, E., Wang, F., Such, F. P., Raso, F., Leoni, F., Tsimpourlas, F., Song, F., von Lohmann, F., Sulit, F., Salmon, G., Parascandolo, G., Chabot, G., Zhao, G., Brockman, G., Leclere. G., Salman, H., Bao, H., Sheng, H., Andrin, H., Bagherinezhad, H., Ren, H., Lightman, H., Chung, H. W., Kivlichan, I., O' O'Connell, I. Osband, I., Gilaberte, C., Akkaya, I., Kostrikov, I., Sutskever, I., Kofman, I., Pachocki, J., Lennon, J., Wei, J., Harb, J., Twore, J., Feng, J., Yu, J., Weng, J., Tang, J., Yu, Candela, J. Q. Palermo, J., Parish, J., Heidecke, J. Hallman, J., Rizzo. J., Gordon, J., Uesato, J., Ward,
Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F, Wei, H., Lin, H., Yang. J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y.. Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report, 2025. httpttps: //arxiv. www./ag./////o.oooooo//.oo...a./ 222222122211 221..////1..11155
Reiter, E. Natural language generation challenges for ex- plainable AI. In Alonso, J. M. and Catala, A. (eds.), Proceedings of the Ist Workshop on Interactive Natural Language Technology for Explainable Artificial Intelli- gence (NL4XAI 2019), pp. 3-7. Association for Compu- tational Linguisties, 2019. doi: 10.18653/11/119-884 URL pttttp:://aclanthhll.oo
J., Huizinga, J., Wang, J., Chen, K., Xiao, K., Singhal, K. Nguyen, K., Cobbe, K., Shi, K., Wood, K., Rimbach, K., Gu-Lemberg, K., Liu, K., Lu, K., Stone, K., Yu, K.,
wrg/wwl////... 0e0p///19-800000 8-802
Shi, F,, Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E.,
12

---

Premisc-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
Schärli, N., and Zhou, D. Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference Machine Learning, ICML'23, JoLR.org, 2023.
Weng, Y., Zhu, M., Xia, F., Li, B., He, S., Liu, S., Sun, B., Liu, K., and Zhao, J. Large language models are better reasoners with self-verification. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Com- putational Linguisties: EMNLP 2023, pp. 2550-2575, Singapore, December 2023, Association for Computa- tional Linguistics. doi: 10. 10.116653/1122023..nnnmm emnlp.167. URL http:: ht htt.:::ss//aaaaatholoooooooo /2023 findiindiiii wew.innnnnnnnnnlgne
Stechly, K., Valmeekam, K. and Kambhampati, S. On the verfficciiintioon limitations of large language models on reasoning and planning tasks, 2024. URL https: //arxiv. org/ abs/ / 2402. 08115.
org
Tian, J., Li, Y., Chen, W., Xiao, L., He, H., and Jin, Y. Di- agnosing the first-order logical reasoning ability through LogicNLI. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3738-3747, Online and Punta Cana, Dominican Re- public, November 2021. Association for Computational Linguistics. doi: 10. 00018663/1112022emmmpp URL hwwpt:tt//htoo nlp-main. 303.
Wu, Y., Jiang, A. Q., Li, W., Rabe, M. N., Staats, C., Jam- nik, M. and Szegedy, C. Autoformalization with large language models, 2022. URL htttttt/////iii.. abs / 2t/p/2225..22 12615.
Wu, Z., Zeng, Q Zhang, Z., Tan, Z., Shen, C., and Jiang, M. Large language models can self-correct with key condition verification. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 12846-12867, Miami, Florida, USA. November 2024. Association for Computational Linguistics. doi: 10.186 53/v 33/11/2024.cmmlpmmain 14. URL https: anlant hology.org/ 2024. emn p-main, . 714/.
org/ em
Tyen, G., Mansoor, H., Carbune, V., Chen, P., and Mak, T. LLMs cannot find reasoning errors, but can correct them given the error location. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 13894- 13908, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1e..1855/111222..02 ngs-acl.826. URL https: ac anthol org /2024. 00002..fnnnae findings-ache 8/
Yang, K., Swope, A. M., Gu, A., Chalamala, R., Song, P., Yu, S., Godil, S., Prenger, R., and Anandkumar, A. Leandojo: Theorem proving with retrieval-augmented language models, 2023, URL https://arxiv: abs 200...16666622
626.
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L.,, Creswell, A., Irving, G., and Higgins, Solv- ing math word problems with process- and outcome- based feedback, 2022. URL 14275.
Yang, K., Poesia, G., He, J., Li, W., Lauter, K., Chaudhuri, S., and Song, D. Formal mathematical reasoning: A new frontier in ai, 2024. URL s/ 22222220 2 16075.
g/
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models, 2023. URL Detps://arxisn org/ abs 200.. 10601.
Wang, X., Hu, Z., Lu, P., Zhu, Y., Zhang, J.. Subramaniam, S., Loomba, A. R., Zhang, S., Sun, Y., and Wang, W. Scibench: Evaluating college-level scientific problem- solving abilities of large language models, 2024. URL caCHqo.
net hor://ormmmmm (ittorummiiuuu1
Yu, L., Jiang, W., Shi, H., Yu, J.. Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. arXiv preprini arXiv:2309. 00099/22220 /2284, 2023.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Cfain-of-thought prompting clicits reasoning in large language models, 2023. URL https: //tp///xivvv////// wtgg/bb///a/ 1903.
Zheng, C., Zhang, 7., Zhang, B., Lin, R., Lu, K., Yu, B., Liu, D., Zhou, J., and Lin, J. Processbench: Identifying process errors in mathematical reasoning, 2024. URL http//bb//222...3
https:
arxiv. //ariii///xxv..
Welleck, S. Liu, J., Bras, R. L., Hajishirzi, H., Choi, Y., and Cho, K. Naturapproofs: Mathematical theorem prov- ing in natural language. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https: //openrevien wwweeeewwwwwwweeeeeeeeeeeeeeeeee o dww.:mmmddddxxxxxdddrd.11..
Zhu, T., Zhang, K., Xie, J., and Su, Y. Deductive beam search: Decoding deducible rationale for chain- of-thought reasoning, 2024. URL https: //arxiv. org/abs/ ab///40/14 11000...76880 17686.
13

---

Premisc-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
A. Appendix
nsteb_number": "has_error"; boolean, "errors":
A.1. Experimental Details
Model For the Llama model (Grattafiori et al., 2024), we used vLLM (Kwon et al., 2023) for model serving and AzureOpenAl for the GPT4o and GPT4-ol (OpenAI, 2024) models. To ensure reproducibility, all generations performed with temperature=0. For all models, we used their instruct variant.
"error_type":
"error_description""
A.2. Data statistics
]
}
This results in total of 607 reasoning chains with 203 posi- tives, 214 negatives, and 190 synthetic negatives. In total,
we have 2,134 steps annotated as Correct, 321 steps as Math- ematical Error, 443 steps as Logical Inconsistency, and 741 steps as Accumulation Error, indicating that native and ac- cumulation errors appear at almost equal rates. Additionally, model each chain-of-thougle (PARC) based the step-level premise annotations. Across 607 PARCs, each chain contains on average 7.30 steps, rep- resenting the length of the chain-of of-thought; 11.27 premises, corresponding to the total premise references across steps; and 10.42 edges linking premises to conclusions. The av- erage depth of the PARC 6.02, measuring the longest path of dependencies, while the maximum width is 1.90, quantifying how many steps can appear the same layer in the PARC. Finally, the branching factor is 1.37, which is the ratio of edges to nodes, indicating that each step typically spawns fewer than two subsequent steps on average.
Chain-Level Error Types
1. "Missing "Missing-Steps"
Definition: Solution lacks crucial concluding steps or final answer derivation
directed acyclic graph
Examples:
Not showing the final calculated value
Missing the ultimate conclusion
Failing to complete the proof
Impact: Makes the solution incomplete or inconclusive
2. "Planning_ Errorr
Definition: The reasoning takes invalid fundamentally incorrect approach
Examples:
Using inapplicable theorems or methods
Solving for incorrect variables
Taking an approach that cannot possibly lead to a solution
A.3. Prompt for Error Annotation with 01
Impact: Makes the entire solution path invalid
Note: Valid but longer approaches (e.g.. integration by parts instead of substitution trick) should NOT be marked errors
You are an expert mathematical reasoning analyzer. Your task to analyze mathematical solutions and generate de- tailed error annotations in specific JSON format. For each solution provided, you must carefully examine the reasoning chain and individual steps to identify any errors or issues.
Step-Level Error Types
1. "Logical Inconsistency"
Response Format
Definition: Steps that violate logical principles or make unjustified conclusions Examples:
Your response must be a valid JSON object following ex-
actly this structure:
False equivalences
Invalid deductions
Unsupported assumptions
verror_ammotations"" {
Note that incorrect use of previous information (exam- ple the step uses a wrong value of a variable) is a Logi- cal Inconsistency Impact: Breaks the logical flow of the solution
"chain_level";
"has_errors"; boolean, "errors":
"error_type": string,
2. "Mathematical-Erroe
"error_descrigtionn: string
Definition: Incorrect calculations, misuse of formulas, or mathematical operations
}
]
Examples:
b, "step_level";
Arithmetic mistakes
Incorrect algebraic manipulations
14

---

Premise-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
GSM8K
MATH
Orca Math
MetaMathQA
Method
Neg
Syn
Pos
Avg.
Nog
Syn
Pos
Avg
Neg
Syn
Pos
Avg
Neg
Syn
Pos
Avg
Full Context
Llama 3.1
48.87
57.68
91.9
58.6
46.91
60.29
90.61
60.2
43.26
51.2
96.7
55.06
50.47
55.15
95.17
59.45
Llama 3.1 70b
58.35
77.03
98.61
71.37
55.55
72.83
96.24
69.77
50.33
64.25
96.74
63.52
55.59
73.13
97.93
69,49
Qwen 2.5 7h
40.88
54.13
100
54.94
40.47
53.60
98.07
56.26
41.20
53.69
100
55.85
43.36
53.18
99.39
56.26
Qwen 2.5 32b
46.76
69.26
99.8
63.56
49.58
66.5
98
65.11
49.44
63.01
99.5
63.06
49.7
67.8
99.6
65.02
Qwen 2.5 72b
47.48
67.17
100
63.11
53.19
67.87
99.79
67.52
50.41
60.33
100
62.50
47.31
63.59
99.05
62.20
GPT4o-mini
52.68
73.14
95.02
66.68
50.41
62.39
93.4
63.03
55.22
61.18
98.6
64,59
51.52
65.42
97.35
64.47
GPT-4o
53.81
75.3
98.33
68.52
50.9
68.41
98.57
66.52
59.55
65.75
100
68.55
56.26
71.72
99.3
69.41
Oracle Premises
Llama 3.1 8b
57.02
82.64
74.61
69.26
54.58
60.94
51.65
56.47
57.43
60.61
59.48
59.09
52.65
68.48
60.51
60.37
Llama 3.1 70b
74.41
85.12
94.69
81,46
68.55
79.88
78.11
74,68
66.78
72.65
93.37
73.45
63.31
83,15
92.16
76.05
Qwen 2.5 7h
52.40
79.37
89.56
69.74
56.86
70.23
79.85
65.96
55.01
66.66
76.65
63.39
56.32
77.06
91.94
70.45
Qwen 32b
66.57
88.64
96.95
80.76
69.24
82.26
89.32
77.98
63.58
69.88
96.31
71.4
59.84
85.74
96.87
76.38
Qwen 2.5 72b
69.48
86.34
98.98
80.58
72.98
81.10
91.03
79.50
69.76
72.77
93.24
74.87
66.23
86.21
98.06
79.50
GPT4o-mini
63.83
85.77
91.29
76.16
65.86
74.99
78.41
71.71
64.03
72.66
83.56
70.74
58.77
85.11
91.44
74.73
GPT-4o
67.03
87.2
94.76
78.74
70.32
77.3
90.65
76.79
71.19
73.69
92.15
75.56
72.76
88.95
94.82
82.88
Model Premises
Llama 3.1 8b
54.61
76.74
65.44
64.53
52.16
59.83
50.96
54.88
55.57
59.82
67.36
59.22
54.52
69.13
62.63
61.78
Llama 3.1 70b
74.51
86.22
94.69
81.92
70.94
79.29
77.91
75.45
64.95
72.28
93.37
72.52
65.69
82,74
89.71
76.52
Qwen 2.5 7h
53.03
78.40
90.37
69.73
58.47
66.87
82.72
65.96
52.07
71.12
84,28
65.38
54.98
73.87
91.02
68.43
Qwen 32b
65.58
86.34
96.95
79.37
68.61
76.96
88.71
75.57
61.89
69.24
95.25
70.26
62.18
85.60
97.32
77.40
Qwen 72b
69.87
85.12
96.38
80.56
69.51
79.92
90.84
77.28
69.45
71.34
94.19
74.41
64.73
85,52
98.12
78.53
GPT4o-mini
58.25
80.81
86.72
72.33
65.84
72.4
71.88
69.45
61.18
72.48
85.94
69.93
57.04
81.63
91.15
72.51
GPT-4o
65.23
86.67
99.76
79.82
70.12
76.67
91.84
76.45
66.33
71.74
93.14
72.96
67.28
88.19
89.85
79.41
Table Detailed results for error identification for dataset and models
Positives
Negatives
Synthetic Negatives
Model Name
Precision
Recall
F1
Precision
Recall
F1
Precision
Recall
F1
Llama 3.1 8b
67.80
89.91
77.31
65.03
91.31
75.96
64.08
86.76
73.71
Llama 70b
83.49
97.09
89.78
80.36
98.87
88.66
79.92
96.68
87.51
Qwen 7b
69.94
62.85
66.21
66.14
67.60
66.86
66.18
60.15
63.02
Qwen 32b
85,98
93.49
89.58
83.78
96.95
89.88
82.45
95.61
88.54
GPT4o-mini GPT-4o
74.93 89.38
84.81
79.56
70.85
87.01
78.10
71.84
86.73
78.59
93.66
91.47
84.97
94.95
89.69
83.43
94.65
88.72
Table Precision, Recall and Fl score for premise identification with GSM8K
Positives
Negatives
Synthetic Negatives
Model Name
Precision
Recall
Precision
Recull
F1
Precision
Recall
F1
Llama 3.1 8b
53,98
82.36
65.22
46.39
82.43
59.37
52.63
82.42
64.24
Llama 3.1 70b
71.13
95.82
81.65
67.82
97.16
79.88
70.46
97.47
81.80
Qwen
59.40
60/30
59.85
48.82
56.02
52.17
59.40
62.87
61.09
Owen 32h
74.57
87.30
80.43
69.98
87.31
77.69
73.13
91.04
81.11
4PT4o-mini GPT-4o
57.58
66.57
61.75
54.42
69.01
60.85
61.18
72.29
66.27
69.76
90.19
78.67
66.95
87.16
75.73
71.48
91.99
80.45
Table 10. Precision, Recall and Fl score for premise identification in MATH
Wrong formula application
Examples:
Note that Mathematical Error can only appear when there is an error in calculation
Using wrong intermediate results
Building upon previously miscalculated values
Impact: Produces incorrect numerical or algebraic results
Impact: Compounds previous mistakes into larger errors
"Accumulation_
Definition: Errors that propagate from previous incorrect
4. "Other"
steps
Definition: Any error that doesn't 't fit into the above cate-
15

---

Premise-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
Positives
Negatives
Synthetic Negatives
Model Name
Procision
Recall
F1
Precision
Recall
F1
Precision
Recall
F1
Ilama 3.1 8h
58.94
84.52
69.45
44.14
76.76
56.05
58.67
82.99
68.74
Llama 3.1 70b
77.82
97.37
86.50
65.18
96.13
77.69
76.29
95.88
84.97
Qwen 7b 58.05
59.09
58.57
48.72
66.48
56.23
59.86
59.41
59.63
Qwen 32b
78.73
91.60
84.68
69.89
88,99
78.29
78.09
92.07
84.50
GPT4o-mini GPT-4o
65.06 79.91
76.92
70.50
51.74
73.81
60.84
65.52
79.37
71.78
92.51
84.68
67.13
88.11
76.26
79.42
90.49
84.62
Table 11. Precision, Recall and Fl score for premise identification with MetaMathQA.
Positives
Negatives
Synthetic Negatives
Model Name
Precision
Recall
Precision
Recall
Precision
Recall
F1
Ilama 3.1 8h
56.02
80.12
65.94
52.60
78.92
63.13
55.31
80.26
65.49
Llama 3.1 70b
72.82
96.83
83.13
68.15
96.42
79.86
72.17
96.93
82.74
Qwen 7b 62.33
61.81
62.07
68.15
96.42
79.86
58.75
59.69
59.22
Qwen 32b
77.73
91.18
83,92
71.55
89.55
79.54
73.55
91.16
81.41
GPT4o-mini GPT-4o
63.24 73.90
75,36
68.77
57.36
73.38
64.39
59.75
73.13
65.77
92.63
83.92
66.97
92.87
77.92
72.50
91.02
80.53
Table 12. Precision, Recall and F1 score for premise identification in OrcaMath
gories Examples:
Focus on correctness rather than elegance or brevity
Notation mistakes
Unclear explanations
Workflow
Formatting issues
Read and understand the problem statement
Impact: Varies depending on the specific error
2. Analyze the reasoning chain step-by-step
Analysis Requirements
3. Check for chain-level errors
1. Examine each step against mathematical principles and theorems
4. Analyze each step for specific errors
5. Verify all premises and justifications
2. Verify all calculations and mathematical operations
6. Ensure completeness of the solution
3. Check for proper use of definitions and formulas
Ensure logical flow between steps
Compare against the provided ground truth answer
A.4. Prompt for Premise Annotation with 01
6. Consider the completeness of the solution
The system prompt and the instruction for the Ol model for identifying premises for a given step is shared in Table 13
Important Notes
Provide ONLY the JSON output, no additional text or explanations
A.S. Prompt for Error identification
Every step in the solution must have a corresponding entry step_ level array
The prompts used for the baseline approach are shared in Appendix 14 and 15 The evaluation for our error identifi- cation with premises are done with the prompts outlined in Tables 16 and 17,
Keep error descriptions clear, specific, and mathematically precise
Use empty arrays for errors when errors exist Ensure your response is always valid ISON that matches the exact format specified
Each error must have both an etror_type and a correspond- ing detailed error_description
Error descriptions should be specific the mathematical
context of the problem
Do NOT penalize valid but verbose approaches (c.g.,
breaking down algebra into multiple steps)
Do NOT mark alternative solution methods as errors unless they are genuinely invalid
16

---

Premisc-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
Instruction:
Given this math word problem and its solution steps, identify the key premises and their relationships.
Problem: (problem[ question"]
"urotlen"qquestion }}
Solution Steps:
{chr(10).joing) jon((())oii(problenn (ssoinppollemsstpsss]]] })}
Return your analysis in this exact JSON format:
[json template]
Critical Rules for Premises:
1. A step NEVER use itself premise. For example, Step 3 cannot any premise labeled [3, "...".
2. Premises can only come from:
-Step 0 (problem statement and fundamental mathematical principles)
Previous steps (steps with lower index)
Any intermediate calculations or logical steps within a single step should be part of that step's reasoning, treated separate premises.
4. Mathematical principles (like properties of operations) should be treated as part of Step 0.
Additional Requirements:
1. Start with Step O containing the problem statement.
For each step after O, copy the EXACT text from the student's ' solution into 'original step'.
Show clear reasoning for how premises lead to conclusions.
4. Return ONLY valid JSON with no additional text.
Do not any special characters like &, i. d etc.
6. Do not add any additional text for formatting it (c.g.. "json"), just output the raw JSON.
7. Maintain the exact same number of steps in the original solution.
Remember:
Each step's premises must strictly come from either the problem statement (Step 0) or previous steps. Never from the current step.
Keep each step atomic-do not split steps into multiple substeps even if they contain multiple calculations.
The number of steps in your output (excluding Step 0) must match exactly with the number of steps in the student's 's solution.
System Prompt:
You are an expert in mathematical reasoning. Your task is analyze solution steps and output JSON object containing:
1. The premises used in each step.
2. The conclusion reached.
3. The reasoning that connects premises to conclusions.
Output MUST be valid JSON with no additional text or explanation.
Table 13. System prompt and instruction for the O1 model to identify premises for given step
17

---

Premisc-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
Instruction:
Question: {question}
Solution far: {solution)
1. "ncinssstncc
Definition: Steps that violate logical principles or make unjustified conclusions
Examples:
False equivalences
Invalid deductions
Unsupported assumptions
Note that incorrect use of previous information (example the step uses wrong value of a variable) is
Logical Inconsistency
Impact: Breaks the logical flow of the solution
"Mathematical "rrrrr"
Definition: Incorrect calculations, misuse of formulas, or mathematical operations
Examples:
Arithmetic mistakes
Incorrect algebraic manipulations
Wrong formula application
Note that Mathematical Error can only appear when there is an error in calculation
Impact: Produces incorrect numerical or algebraic results
"Accumulation "Errorr
Deeiiition: Errors that propagate from previous incorrect steps
Examples:
Using wrong intermediate results
Building upon previously miscalculated values
Impact:
Compounds previous mistakes into larger errors
4. "Other"
Definition: Any error that doesn't 't fit into the above categories
Examples:
Notation mistakes
Unclear explanations
Formatting issues
Impacc: Varies depending on the specific error
Statement analyze: step
Format your response as: Reasoning: [detailed analysis of the statement's 's validity] Verdict: [CORRECT, Mathematical Error, Logical Inconsistency, or Accumulation Error]
Table Baseline error identificatoon Instruction
18

---

Premisc-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
System Prompt:
You are a helpful Al assistant that analyzes mathematical solution steps. Your task is to determine each statement COMPLETELY correct by carefully analyzing its validity. Focus ONLY on whether the current step is valid do consider whether it helps reach the final answer or whether better steps could have been taken. Mark statement CORRECT unless you find a specific
Table 15. Baseline error identification System prompt
System Prompt:
Your task is to determine whether given sentence contains any mathematical errors. For mathematical error, check the sentence contains mathematical calculations (arithmetic or algebraic), and whether they are incorrect. If there are such errors, mark the sentence as "mathematical error" Mathematical errors can only come from incorrect results of mathematical operations If no such errors exist, mark it "correct".
Note: mathematical error can only come from incorrect numerical or algebraic calculations (i.e. wrong multiplication, wrong addition etc.) if there are no numerical or algebraic calculations done, you can mark as correct
Instruction :
Statement analyze:
step
Format your response as:
Reasoning: [detailed analysis of the statement's validity]
Verdict: [correct or mathematical.error/
Table 16. Math error identification instruction
System Prompt:
You are provided with math question, a statement which a step in the solution to the question and the premises this steps (the question is also a premise). Your task is identify whether the step follow naturally from the premises not. If the current step contradicts the premises, mark is logical iigiiaalicconssttenny If the step be directly inferred from the premises, mark correct. You should not check whether the premises are correct, assume they are correct. Only check the sentence given.
Instruction :
Given Premises:
Question: {question}
Previous steps as premise: (premises)
Statement to analyze: {step}
Guidelines:
1. for incoossstcnnsiseec check if the step was performed under misinterpretation of the premises, made invalid deductions or had unsupported assumptions
Don't check for correctness of the premises, your only task to check correctness of given sentence
Format your response Reasoning: [detailed analysis of the statement's validity] Verdict: [correct, )
Table 17. Logical error identification instruction
19

---

Premisc-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs
Instruction:
You are provided with question, a partial solution, and the next step in the solution. Your task is to identify the steps that serve as premises for the given next step. step qualifies as a premise the next step directly relies on information from that step. Based on the identified premises, the correctness of the next step should be fully verifiable. Question (Step 0): {question]
Solution far: {solution]
Next step to analyze: {step}
For the step above, identify which previous steps (including Step 0 the question) are premises and explain why each one is necessary. Remember:
1. A step cannot be premise itself
2. The question (Step 0) can be premise if used directly Generate ONLY the premises and nothing else.
Format your response with one premise per line as:
Step X: (explanation of why this step is necessary for the current step]
Table 18. Prompt for evaluating models in the premise identification task (zero shot)
20


Reviews:
[{"review_id": "bc9DK65d9y", "reviewer": "Reviewer_XAi3", "summary": "This paper introduces a new category of errors (accumulation errors) and Premise-Augmented Reasoning Chains (PARC) as a method to improve error identification in mathematical reasoning with Large Language Models (LLMs). To evaluate this method, the authors construct PERL (Premises and ERrors identification in Language models), a benchmark dataset containing annotated premise links and error types in mathematical reasoning chains. Their results demonstrate that LLMs can achieve ≥90% recall in premise extraction, highlighting the effectiveness of their method.\n\n\n## update after rebuttal\nI'd like to thank the authors for their response and keep my positive rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their kind remarks on the novelty and thorough experimental design. Here we address the concerns\n\n**concern 1** - low precision of premise identification and mapping\n\n**response** - We would like to highlight that our claim is originated by the high recall primarily. In the context of error identification, verifiability (recall) is more important than minimality (precision), since missing even a single premise can compromise the step’s and subsequent steps' verifiability. Further, upon closer inspection we observe that, low precision often results from the ground truth premise set being very small. For example, if step 7 has ground truth premises {1} and the model predicts {0,1}, precision is 50%, even though the context remains well-pruned. \n\n**concern 2** - lack of theoretical guarantees \n\n**response** - We primarily take an empirical route, and as already pointed out by you, we do thorough experiments across model families, scale and datasets to prove robustness of our method. We hope that the released dataset can help the community to  foster more research in this direction as well.\nTo further provide proof that our method is effective we provide results on the popular Processbench dataset, which has step level annotations done by humans. **Please refer to our response to concern 1 raised by Reviewer kjS7 for the Processbench results.**\n\n\n**concern 3** -  Mathematical Error and Logical Inconsistency are merged\n\n**response** - When we say that the boundary between them is “thin”, we imply that even a simple mathematical calculation (or manipulation) could easily be considered as a logical error (due to fault in mathematical logic). However we agree that for completeness, it's essential to have a detailed breakdown per error category, which we present here. ( we will include these in the final version of the paper)\n\n\nGSM8k\n| Model            | Context        | Logical Error | Mathematical Error |\n|------------------|----------------|---------------|--------------------|\n| Llama 3.1 8b     | Full context   | 29.9          | 41.1               |\n|                  | Model premises | 55.6          | 64.9               |\n| Llama 3.1 70b    | Full context   | 48.5          | 72.4               |\n|                  | Model premises | 79.6          | 64.5               |\n| GPT4o-mini       | Full context   | 33.4          | 58.6               |\n|                  | Model premises | 64.7          | 65.8               |\n| GPT-4o           | Full context   | 44.4          | 52.4               |\n|                  | Model premises | 64.1          | 74.0               |\n| Qwen 7b          | Full context   | 21.4          | 20.7               |\n|                  | Model premises | 59.0          | 39.1               |\n| Qwen 72b         | Full context   | 31.7          | 44.5               |\n|                  | Model premises | 69.9          | 64.6               |\n\n \nMATH\n \n| Model            | Context        | Logical Error | Mathematical Error |\n|------------------|----------------|---------------|--------------------|\n| Llama 3.1 8b     | Full context   | 46.4          | 45.1               |\n|                  | Model premises | 57.9          | 63.7               |\n| Llama 3.1 70b    | Full context   | 46.0          | 75.0               |\n|                  | Model premises | 79.0          | 66.3               |\n| GPT4o-mini       | Full context   | 54.1          | 61.5               |\n|                  | Model premises | 83.1          | 67.4               |\n| GPT-4o           | Full context   | 55.1          | 59.4               |\n|                  | Model premises | 63.4          | 64.2               |\n| qwen 7b          | Full context   | 27.7          | 34.7               |\n|                  | Model premises | 77.6          | 49.2               |\n| qwen 72b         | Full context   | 47.8          | 63.8               |\n|                  | Model premises | 77.7          | 66.8               |\n \n\n\n**concern 4** - Graph-of-Thoughts [1] also disc…\n\n**response** - Thanks for raising this. Graph of thought induces structure at inference time, while we do it after inference, and later use that to improve error identification. But there is definitely a resemblance between them, and we will include them in our camera ready version.", "claims_and_evidence": "The authors claim that “off-the-shelf LLMs can detect premises for a given step with high accuracy for mathematical reasoning.” However, my main concern lies in the low precision of premise identification and mapping, as reported in Tables 1 and 2. While I acknowledge the high recall achieved by LLMs, precision remains a critical factor, particularly given the minimality requirement defined in Lines 151–164—where a premise set should be minimal such that removing any element renders the corresponding step unverifiable. With precision ranging from 60% to 80%, the identified premises do not seem to consistently meet this minimality criterion, leaving the authors' claim insufficiently substantiated.", "methods_and_evaluation_criteria": "The PARC framework is well-motivated and introduces an intuitive premise-based verification process for mathematical reasoning. The PERL dataset encompasses a diverse set of math word problems from GSM8K, MATH, Orca-Math, and MetaMathQA, ensuring a broad range of difficulty levels. Additionally, the chosen evaluation metrics—precision, recall, and F1 for premise detection, along with accuracy for error identification—are appropriate for assessing the effectiveness of the proposed approach.", "theoretical_claims": "The formalization of premise extraction and mapping problem (Section 3.1) is reasonable. However, the paper lacks in-depth theoretical guarantees or analysis on the solution quality of the proposed algorithm (Algorithm 1), leaving open questions about its optimality and robustness in premise identification.", "experimental_designs_or_analyses": "The paper presents a thorough experimental design, with evaluations conducted across multiple datasets such as GSM8K and MATH, ensuring a diverse assessment of the proposed method. However, a major concern lies in the experimental setup described in Section 5.2, particularly in Lines 373–378, where the authors state that Mathematical Error and Logical Inconsistency are merged into a single error type, Error, due to their “thin boundary”. Since these are well-established and distinct error types, the inability of models to differentiate between them suggests a limitation in model capacity rather than an inherent issue with error categorization. Merging these categories appears to artificially simplify the task, potentially leading to an ad-hoc and unfair evaluation of performance. A more rigorous analysis preserving the original error distinctions would provide a clearer assessment of the model’s reasoning capabilities.", "relation_to_broader_scientific_literature": "This work introduces a new error taxonomy by defining accumulation errors and situates itself within existing research on reasoning verification. While building on prior work, it presents a novel structured verification and error detection approach, contributing a more systematic method for identifying and analyzing reasoning errors in math reasoning tasks.", "essential_references_not_discussed": "Graph-of-Thoughts [1] also discusses non-linear reasoning structures, which relate to PARC’s DAG-based structure but is not cited. A discussion would be helpful.\n\n[1] Besta, Maciej, et al. \"Graph of thoughts: Solving elaborate problems with large language models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 16. 2024.", "other_strengths_and_weaknesses": "See previous sections.", "other_comments_or_suggestions": "See previous sections.", "questions_for_authors": "See previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Im4eo0b9Nv", "reviewer": "Reviewer_gQAd", "summary": "This paper studied the step-level verification of CoT reasoning, and proposed a PARC framework that converted linear reasoning chain into DAG by introducing premise links. Based on the framework, the authors defined a new error type named accumulation error, and constructed PERL dataset to evaluate the framework. The experiments demonstrated that PARC helped to identify step-level errors in CoTs.\n## update after rebuttal\nThe responses have addressed most of my concerns. After reading other reviews and authors' responses, I decide to keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their kind remarks on the novelty and thorough experimental setup of our method. Here we try to address their concerns.\n\n**concern 1** - Since the premise is based on the step, it would be better to make it clearer what is an intermediate step defined, and how to get the step list from a complete CoT.\n\n**response** -  In our work, we explicitly prompt the generator model to answer in a step by step format, with formatting instructions in the prompt. The generated solution looks like \nStep 1: …\nStep 2: …\nFinally we do a simple regex to extract the steps. \n\n**concern 2** - How did the authors ensure or evaluate whether the premise satisfied the three properties in page 3.\n\n**response** - In our work, we prompt the highly capable O1-preview model to generate the ground truth premises and later manually verify the generated premises on whether they satisfy the verifiability and minimality conditions.\n\n\n**concern 3** - What’s the necessity of the proposed accumulation error?\n\n**response** - A holistic evaluation of LLM reasoning should consider the entire reasoning chain rather than relying solely on a binary correct/incorrect outcome of the final answer. Reasoning chains may contain subtle intermediate errors despite following a globally correct plan, ultimately rendering the solution incorrect, a nuance overlooked by final-answer correctness metrics. Prior works like PRM800K and ProcessBench have typically annotated reasoning chains only up to the first erroneous step, discarding subsequent steps due to ambiguity. To our knowledge, we are the first to formally introduce the concept of accumulation errors, enabling a more comprehensive evaluation of reasoning chains. Similar to how teachers award partial credit for nearly correct answers, evaluation frameworks should recognize when the overall reasoning plan is sound despite minor mistakes, assigning partial credit accordingly. Accumulation errors, where a step is locally correct but built on flawed premises, explicitly capture this subtlety. Identifying accumulation errors highlights how earlier mistakes compromise the reliability of the reasoning chain, making it essential to incorporate these errors into holistic scoring methods.\n\n**concern 4** - it may be problematic to treat the CoT as correct based on the final answer, due to possible step errors as studied in the paper.\n\n**response** - \nWe completely agree with the reviewer on this - in our work, we observed that even in the positive reasoning chains, there were a few steps that were annotated as error by the O1 model, indeed LLMs suffer from False Positives, which is established from the work - https://arxiv.org/pdf/2502.06217 . \nIn GSM8k - 4 false positives out of 50\nIn MATH - 4 out of 50\nMetamathqa - 5 out of 50\nOrcamath - 4 out of 50\nWhen we release our dataset, we will make sure to flag them as false positives.", "claims_and_evidence": "The claims are well supported by the experimental results. The authors specially constructed several datasets and conducted extensive experiments to demonstrate the effectiveness of the proposed framework, including the premise identification and error identification.", "methods_and_evaluation_criteria": "Overall the proposed method works for CoT verification. However, there are still some issues that are not clearly explained in method design.\n1. Since the premise is based on the step, it would be better to make it clearer what is an intermediate step defined, and how to get the step list from a complete CoT.\n2. How did the authors ensure or evaluate whether the premise satisfied the three properties in page 3. And how to evaluate whether a step is verifiable.\n3. What’s the necessity of the proposed accumulation error. According to the paper, if the error exists, it means that there are native errors, which is enough to indicate the incorrect reasoning.", "theoretical_claims": "There is no theoretical claim in the paper.", "experimental_designs_or_analyses": "The experiments are extensive and solid. I have one concern that in dataset construction, it may be problematic to treat the CoT as correct based on the final answer, due to possible step errors as studied in the paper.", "relation_to_broader_scientific_literature": "The paper focused on the verification of the CoT based on premise DAG. Compared with existing works, the authors proposed an automatic premise recognition method without relying on predefined CoT template and hurting reasoning, and propose a novel error type focusing on error propagation.", "essential_references_not_discussed": "Related works are well cited.", "other_strengths_and_weaknesses": "In summary, the strengths of the paper are listed as follows.\n\n1. The paper proposed a novel premise DAG-based method PARC for step-level verification of CoT reasoning.\n2. The authors conducted extensive experiments to demonstrate the effectiveness of the framework.\n\nThe weakness of the paper are as follows.\n\n1. Some details of method design are unclear in the paper. See the \"Methods\" and \"Experiments\" parts.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. How did the authors ensure or evaluate whether the premise satisfied the three properties in page 3. And how to evaluate whether a step is verifiable.\n2. What’s the necessity of the proposed accumulation error. As the native errors are enough to indicate the incorrect reasoning.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ovyvvQmAxt", "reviewer": "Reviewer_K1LJ", "summary": "The authors explore how to improve error identification in reasoning chains, which consist of multiple individual reasoning steps. They start by converting the reasoning chain into a directed acyclic graph, called Premise-Augmented Reasoning Chains (PARC), where the nodes that are reasoning steps and the edges indicate the dependency on premises from previous reasoning steps. They present two LLM-based approaches: providing the full reasoning chain until the current step and asking the LLM to identify the premises of the current step and alternatively building pairs of the current step with each of its predecessors and asking the LLM, whether a given step is the premise of the current step. Additionally to mathematically errors within a reasoning step and logical inconsistencies, where a reasoning step is not consistent with its premises, the authors also propose a new error type: the accumulation error, where the reasoning itself is correct, but depends on incorrect premises. The authors also derive a dataset, called Premises and ERrors identification in Language models (PERC), which they intend to publish, that is based on 607 positive, negative and synthetic negative samples, which they subsequent use to evaluate seven different models. They find that LLMs can identify the premises reasonably well and that their approach improves error identification.\n\n## update after rebuttal\n\nWhile the rebuttals answered some of my questions, the addition of another dataset has not convinced me enough to raise my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their appreciation of the writing and experiments. Here we address their concerns\n\n**Typos** - Thanks for bringing those to our notice, we will definitely fix those in the final version.\n\n**concern 1** - Dataset is a bit small, no external datasets used\n\n**response** - \nTo further provide proof that our method is effective we provide results on the popular Processbench dataset, which has step level annotations done by humans. \n\nGsm8k (400 examples)\n| Model      | Method   | Correct acc | Wrong acc | F1   | Δ (Delta) |\n|------------|----------|-------------|-----------|------|-----------|\n| Llama 8B   | Baseline | 17.1        | 36.7      | 23.3 | –         |\n|            | Ours     | 33.7        | 37.8      | 35.6 | 12.3      |\n| Llama 70B  | Baseline | 77.7        | 57.5      | 66.1 | –         |\n|            | Ours     | 89.6        | 70.0      | 78.6 | 12.5      |\n| Qwen 7B    | Baseline | 66.3        | 36.7      | 47.2 | –         |\n|            | Ours     | 60.1        | 38.6      | 47.0 | -0.2      |\n| Qwen 32B   | Baseline | 97.9        | 43.0      | 59.8 | –         |\n|            | Ours     | 95.9        | 55.1      | 70.0 | 10.2      |\n| Qwen 72B   | Baseline | 98.4        | 61.4      | 75.6 | –         |\n|            | Ours     | 97.8        | 59.7      | 74.1 | -1.5      |\n\n\nMath (1000 examples)\n\n| Model      | Method   | Correct acc | Wrong acc | F1   | Δ (Delta) |\n|------------|----------|-------------|-----------|------|-----------|\n| Llama 8B   | Baseline | 5.6         | 19.1      | 8.7  | –         |\n|            | Ours     | 11.0        | 27.5      | 15.7 | 7.0       |\n| Llama 70B  | Baseline | 32.4        | 32.8      | 32.6 | –         |\n|            | Ours     | 61.6        | 55.4      | 58.3 | 25.7      |\n| Qwen 7B    | Baseline | 46.0        | 25.4      | 32.7 | –         |\n|            | Ours     | 45.6        | 41.2      | 43.3 | 10.6      |\n| Qwen 32B   | Baseline | 90.0        | 22.4      | 35.9 | –         |\n|            | Ours     | 86.9        | 53.9      | 66.5 | 30.7      |\n| Qwen 72B   | Baseline | 88.5        | 33.7      | 48.8 | –         |\n|            | Ours     | 86.7        | 53.9      | 66.5 | 17.7      |\n\n\n**concern 2** -  the baseline performance of these models is missing\n\n**response** - In our paper we have two tasks - premise identification and error identification. The former is a novel task in itself, hence we don’t explicitly compare against any baseline as such. Instead we compare the aggregative and dyadic approaches as an ablation to how premises can be identified. For the error identification task, the baseline is when the entire context is fed into the LLM as compared to our approach where we only use the premises generated (which is a standard LLM as a judge scenario, please refer to the Table 3 the rows tagged \"full context\" is the baseline here). We would also like to highlight that existing popular frameworks like Roscoe and Receval assign a chain level scores, and hence are not directly comparable to our use case. \n\n\n**concern 3** - discussion of the additional compute resources, time and costs\n\n**response** - As we query the judge LLM twice we observe a 2X latency as compared to the baseline. We empirically verified this for the Qwen32b model, across all 4 datasets.\n\n\n**question 1** - Why do longer reasoning chains make it harder to verify individual steps?\n\n**response** - The longer a reasoning chain gets, the longer the context is for the later steps. This further implies that a lot of irrelevant information (non-premises) is fed into the context, making it hard for the model to reason whether the step was correct or not. \n\n\n**question 2** - If the nodes are the reasoning steps, how can the edges link to premises?\n\n**response** - In our formulation in the DAG - each node is a step, and directional edges capture dependencies between steps (nodes). More specifically, if a step i is premise to step j, there is a step (i->j) \n\n\n**question 3** - Why are there more PARCs than chains in the dataset? (A2)\n\n**response** - Indeed it should be 607 and we will correct this in our final paper", "claims_and_evidence": "The claims are reasonably support by their evaluation.", "methods_and_evaluation_criteria": "I think the methodology is sound in general and the authors evaluate seven models, showing that their approach at least generalizes in this dimension. The chosen metrics seem appropriate.\n\n* The benchmark seems a bit small: only 607 datapoints.\n* No external dataset used, so it is unclear how much their approach would generalize.", "theoretical_claims": "Yes, to a certain degree, I have checked their formulas, which are reasonably most of the time, but contain some errors as well some notation mistakes.\n\n* line 143, second column: I believe the capital R should be a lowercase r to be consistent.\n* line 202: wrong symbol, if my understanding is correct - should be I instead of F\n* algorithm 1: input and output of algorithm should be a small r?", "experimental_designs_or_analyses": "The experiments are reasonably well designed, albeit follow their approach very much. The author provide some end-to-end results, but only for their own dataset. The authors supply some form of ablation study, where they test different approaches.\n\n* If my understanding is correct, the baseline performance of these models is missing, making it harder to judge by how much the whole approach improves upon the base models.\n* No discussion of the additional compute resources, time and costs.", "relation_to_broader_scientific_literature": "The ideas are for the most part novel. The use of the DAG to model the dependencies between thoughts/reasoning step is somewhat akin to Graph of Thoughts, however is here applied after the generation is finished instead of during the generation, which is then used to identify errors.", "essential_references_not_discussed": "None to my knowledge.", "other_strengths_and_weaknesses": "The writing is clear and the paper is easy to follow. While the evaluation is detailed to a certain degree, for example the number of evaluated models, other aspects such as external datasets or performance of baseline models are missing.", "other_comments_or_suggestions": "* I suggest to also capitalize \"Reasoning\" in the title.\n* line 16, second column: \"(CoT; Wei et al. (2023))\" - wrong cite command\n* line 111: \"section 5\" - \"Section\" should be capitalized to be consistent\n* line 143, second column: I believe the capital R should be a lowercase r to be consistent.\n* 3.1/3.2/3.3: steps s are sometimes bold and sometimes not \n* line 250/251: \"Algorithm~1\" should be on a single line\n* line 254: should be \"Tables\" instead of \"Appendix\" (or the correct Appendix A.5 should be referred to) \n* 5.1: recall is sometimes used capitalized and sometimes not \n* table 3, caption: \"Error identification\" - should not be capitalized; \"Premises\" -> \"Model Premises\" to be consistent\n* line 410: \"Identification :\" - extra white space before the colon\n* table 4, caption: \"Error identification\" - should not be capitalized\n* table 5: unclear from the caption what data(set) is shown\n* line 406/407, second column: \"Table~3\" - should be on a single line\n* line 414/415, second column: \"Table~4\" - should be on a single line\n* references:\n  * \"Alexa arena: A user-centric interactive platform for embodied ai\" - at least abbreviations should be capitalized; URL cuts into the margin\n  * \"OpenAI o1 System Card\" - cited twice\n  * I suggest to cite the conference versions for example CoT (NeurIPS'22) or ToT (NeurIPS'23)\n  * \"Metamath: Bootstrap your own mathematical questions for large language models\" - cited differently than the other arXiv papers\n* line 853: should be \"Tables\" instead of \"Appendix\"\n* line 1013: missing brackets around \"step\"", "questions_for_authors": "* Why do longer reasoning chains make it harder to verify individual steps?\n* If the nodes are the reasoning steps, how can the edges link to premises?\n* Why are there more PARCs than chains in the dataset? (A2)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nANZy4pTaB", "reviewer": "Reviewer_kjS7", "summary": "This paper looks into the problem of reference-free verification of LLM reasoning chains in the context of mathematical reasoning. Authors hypothesize that a step in a reasoning chain should be verified only under its premises, and propose constructing Premise-Augmented Reasoning Chains (PARC) to improve the traceability of a reasoning chain. Authors create a corresponding dataset, called PERL (Premises and ERrors identification in Language models) to test LLMs’ capability in identifying premises as well as the effectiveness of premises.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their kind remarks on the writing and experimental details of the paper. Here we address their concerns\n\n**concern 1** - Study is limited to the four popular mathematical datasets, and three commonly used LLM\n\n**response** - Please note that we used three popular model families, and for each family we included at least 2 models of different scales, making the number of models=6. Which establishes the robustness of our method across models. Further here we attach some additional experiments done on the recently released processbench dataset for the error identification task, where across model classes we see significant improvements in error identification, further showcasing our method’s reliability\n\nGsm8k\n| Model      | Method   | Correct acc | Wrong acc | F1   | Δ (Delta) |\n|------------|----------|-------------|-----------|------|-----------|\n| Qwen 7B    | Baseline | 66.3        | 36.7      | 47.2 | –         |\n|            | Ours     | 60.1        | 38.6      | 47.0 | -0.2      |\n| Qwen 72B   | Baseline | 98.4        | 61.4      | 75.6 | –         |\n|            | Ours     | 97.8        | 59.7      | 74.1 | -1.5      |\n| Qwen 32B   | Baseline | 97.9        | 43.0      | 59.8 | –         |\n|            | Ours     | 95.9        | 55.1      | 70.0 | 10.2      |\n| Llama 8B   | Baseline | 17.1        | 36.7      | 23.3 | –         |\n|            | Ours     | 33.7        | 37.8      | 35.6 | 12.3      |\n| Llama 70B  | Baseline | 77.7        | 57.5      | 66.1 | –         |\n|            | Ours     | 89.6        | 70.0      | 78.6 | 12.5      |\n\n\nMath\n\n| Model      | Method   | Correct acc | Wrong acc | F1   | Δ (Delta) |\n|------------|----------|-------------|-----------|------|-----------|\n| Qwen 7B    | Baseline | 46.0        | 25.4      | 32.7 | –         |\n|            | Ours     | 45.6        | 41.2      | 43.3 | 10.6      |\n| Qwen 72B   | Baseline | 88.5        | 33.7      | 48.8 | –         |\n|            | Ours     | 86.7        | 53.9      | 66.5 | 17.7      |\n| Qwen 32B   | Baseline | 90.0        | 22.4      | 35.9 | –         |\n|            | Ours     | 86.9        | 53.9      | 66.5 | 30.7      |\n| Llama 8B   | Baseline | 5.6         | 19.1      | 8.7  | –         |\n|            | Ours     | 11.0        | 27.5      | 15.7 | 7.0       |\n| Llama 70B  | Baseline | 32.4        | 32.8      | 32.6 | –         |\n|            | Ours     | 61.6        | 55.4      | 58.3 | 25.7      |\n\n**concern 2** - how large is the \"other\" group, and what's inside\n\n**response** - We observed that the frequency of the other error type in our ground truth dataset was quite low, where only 10 steps were marked as Other. Upon manual inspection, it revealed that the other group is manifested in the forms of - “The solution becomes stuck in a repetitive loop” and “Step is incomplete ”.\n\n**concern 3** - In particular, it would be interesting to see how each aspect - mathematical error, logical inconsistencies, and accumulation errors - affects final error detection\n\n**response** - Since we are restricted by 5000 characters, we request the reviewer to kindly see the response to the third concern by reviewer xai3, where we shared the detailed numbers. \n\n**concern 4** - Authors highlight the importance of newly introduced accumulation errors, but do not provide evidence in their study\n\n**response** - A holistic evaluation of LLM reasoning should consider the entire reasoning chain rather than relying solely on a binary correct/incorrect outcome of the final answer. Reasoning chains may contain subtle intermediate errors despite following a globally correct plan, ultimately rendering the solution incorrect—a nuance overlooked by final-answer correctness metrics. Prior works like PRM800K and ProcessBench have typically annotated reasoning chains only up to the first erroneous step, discarding subsequent steps due to ambiguity. To our knowledge, we are the first to formally introduce the concept of accumulation errors, enabling a more comprehensive evaluation of reasoning chains. Similar to how teachers award partial credit for nearly correct answers, evaluation frameworks should recognize when the overall reasoning plan is sound despite minor mistakes, assigning partial credit accordingly. Accumulation errors, where a step is locally correct but built on flawed premises, explicitly capture this subtlety. Identifying accumulation errors highlights how earlier mistakes compromise the reliability of the reasoning chain, making it essential to incorporate these errors into holistic scoring methods.", "claims_and_evidence": "Through extensive experiments on Math datasets, authors show that verifying each step under its corresponding premises increases the accuracy of identifying errors and their types. In their error design, authors consider 3 error types specific to mathematical reasoning: mathematical error, logical inconsistencies, and accumulation errors. Authors highlight the importance of newly introduced accumulation errors, but do not provide evidence in their study (i.e. through the ablation study).", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Study is limited to the four popular mathematical datasets, and three commonly used LLM. Results are consistent across experiments.", "relation_to_broader_scientific_literature": "Authors elaborate on existing research, generalizing previously developed error taxonomy (i.e. Roscoe), and extensive work on using Verifiers to improve model's reasoning abilities.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Overall, paper is clear and well written. As LLMs are often used as Judges and Verifiers, it is important to know how good they actually are at this task, how can we meta-evaluate their evaluation skills, and improve on them. Proposed approach can be used to enhance  evaluations of the mathematical reasoning. In my opinion, this paper could benefit from additional ablation study.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1.  You compare with baseline that classifies each reasoning step according to the predefined error taxonomy. In reality, there can be other error types outside those you pre-defined. I have noticed you did have \"Other\" error type in the baseline prompt (Table 12, p18). Do you have any statistics about the final error distribution within each dataset in the baseline setup? how large is the \"other\" group, and what's inside?\n2. Paper could benefit from additional ablation study that would highlight the importance of the proposed taxonomy. In particular, it would be interesting to see how each aspect - mathematical error, logical inconsistencies, and accumulation errors - affects final error detection.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a method to construct premise-augmented reasoning chains, which are useful in increasing the verification capabilities of language models. It also provides a way to quantify the accumulation kind of error in the target model's reasoning. Reviewers appreciated the motivation of the framework and its application to math benchmarks. The addition of another benchmark, ProcessBench during the rebuttal was useful. 
Overall, the paper presents a useful idea to improve the capability of LLMs to evaluate reasoning.
Decision: Accept (poster)