RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Authors: Xinnuo Xu
Rachel Lawrence
Kshitij Dubey
Atharva Pandey
Risa Ueno
Fabian Falck
Aditya V. Nori
Rahul Sharma
Amit Sharma
Javier Gonzalez
Venue: ICML 2025 poster (2025)
Keywords: Reasoning
Large Language Models
Causality
Evaluation
Benchmarks
Primary area: general_machine_learning->evaluation

Abstract: Recent Large Language Models (LLMs) have reported high accuracy on reasoning benchmarks. However, it is still unclear whether the observed results arise from true “reasoning” or from statistical recall of the training set. Inspired by the ladder of causation (Pearl, 2009) and its three levels (associations, interventions and counterfactuals), this paper introduces RE-IMAGINE: a framework to characterize a hierarchy of reasoning ability in LLMs, alongside an automated pipeline to generate problem variations at different levels of the hierarchy. By altering problems in an intermediate symbolic representation, RE-IMAGINE generates arbitrarily many problems that are not solvable using memorization alone. Moreover, the framework is general and can work across reasoning domains, including math, code, and logic. We demonstrate our framework on four widely-used benchmarks to evaluate several families of LLMs, and observe reductions in performance when the models are queried with problem variations. These assessments indicate a degree of reliance on statistical recall for past performance, and open the door to further research targeting skills across the reasoning hierarchy.

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Xinnuo Xu
Rachel Lawrence" Aditya V. Nori!
Kshitij Dubey *2 Atharva Pandey" 22 Risa Ueno
Fabian Falck'
Rahul Sharma
Amit Sharma?
Javier Gonzalez'
Abstract
dented results on reasoning benchmarks, e.g. FrontierMath (Glazer et al., 2024) and ARC-AGI (Chollet et al., 2024), With growing reliance on LLMs across wide-ranging appli- cations, it is increasingly important to clarify the strengths and limitations of these apparent reasoning abilities.
Recent Large Language Models (LLMs) have re- ported high accuracy reasoning benchmarks. However, it is still unclear whether the observed results arise from true "reasoning" or from statisti- cal recall of the training set. Inspired by the ladder of causation (Pearl, 2009) and its three levels (as- sociations, interventions and counterfactuals), this paper introduces RE-IMAGINE: framework characterize hierarchy of reasoning ability in L.MMs, alongside a scalable pipeline to generate problem variations across all the levels of the hi- erarchy. By altering problems in an intermediate symbolic representation, RE-IMAGINE generates arbitrarily many problems that are not solvable using memorization alone. The framework is gen- eral and work across reasoning domains, in- cluding math, code, and logic. We demonstrate the type of insights that RE-IMAGINE can gener- ate on four widely-used benchmarks, which we use to evaluate reasoning on several families of LLMs. We observe reductions in performance when the models are queried with problem vari- ations. These assessments indicate degree of reliance statistical recall for past performance, and open the door to further research targeting skills across the reasoning hierarchy.
Reasoning is a cognitive process. It involves using facts or premises to make inferences about conclusions judgments (Holyoak & Morrison, 2005). In the realm of LLMs and AI, reasoning is understood to be the ability of model to demonstrate logically correct systematic processes that surpass mere statistical pattern recognition in the training set (González Nori, 2024).
Traditionally, the evaluation of reasoning in LLMs has been focused on their performance across fixed benchmarks in domains such as math (Cobbe et al., 2021), programming (Wu et al., 2024; Gu et al., 2024), real-world logic (Jin et al., 2023a) and others (Wang et al., 2019; Hendrycks et al., 2020; Srivastava et al., 2022). However, debate persists on whether the observed results occur from genuine reasoning or from mere statistical recall of training data (Mitchell & Krakauer, 2023) particularly for training data which, in the case of published benchmarks, may have information leakage from the test set (Zhou et al., 2023). Defining principled ways to make this distinction is crucial for advancing AI and controlling potential hazards and risks (Weidinger et al., 2021).
Recently, several surveys have explored how to evaluate reasoning beyond memorization in LLMs (Xu et al., 2025; Huang & Chang, 2023). In general, two main approaches have emerged. One aims to develop novel reasoning tasks such as mystery blocksworld (Webb et al., 2024), ARC- AGI (Chollet al., 2025), and others (Zhu et al., 2023). An alternative to create novel variations of existing bench- marks, e.g. for math (Mirzadeh et al., 2024; Srivastava et al., 2024), analogies (Lewis & Mitchell, 2024), and diverse tasks across code, math, and logic (Wu et al., 2023; Zhang et al., 2024). common strategy for creating such variations involves leveraging symbolic representations of problems such as functional templates (Mirzadeh et al., 2024; Srivas- tava et al., 2024), reasoning or causal graphs (González & Nori, 2024; Hüyük et al., 2024; Yang et al., 2024), planning tasks (Valmeekam et al., 2022) or code (Li et al., 2024).
1. Introduction
Recent advancements in Artificial Intelligence (AI) have sparked increasing interest in the development of reasoning systems. Central to this goal are Large Language Models (LLMs)- models like OpenAl's ol (Jaech et al., 2024), o3 (OpenAI, 2024), or DeepSeek-RI R1 (Team, 2025) show com- plex problem-solving abilities, and demonstrate unprece-
'Equal contribution 'Microsoft Research Cambridge, UK Miirosooft Research India. India. Correspondence to: Xinnuo Xu <xinnuoxue @ microsoft.com>>, Rachel Lawrence <rachel wwwchellaaeenceeemrrrofoocommm microsoft.com...
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Level
Description
Examples
Evaluation metric(s)
References
Cobbe al. (2021)
Problems in GSM8K. CI Cldderr, CRUXEval Loop
Jin et al. (2023a)
1. Observe
Original problem.
Task performance (original).
Kamath et al. (2024)
Gu et al. (2024)
Replacing numeric value, changing variable name. modifying operator, irrelevant information
Mirzadeh et al. (2024)
Mutated problem by replacing or adding components.
Task performance (original and mutated).
Srivastava al. (2024)
2. Mutate
Wu et al. (2023)
Lewis & Mitchell (2024)
Original problem augmented with 'maggine' statement, modifying the original statements assertions hefore it.
Task performance (original and augmented)
Extra logic involving revisions counterfactual statements.
González & Nori (2024) Hüyük al. (2024)
3. Imagine
PN and PS (only in counterfactuals)
Table 1. Hierarchy of problem variations introduced in the RE- DE-IMAGINE framework to evaluate LLMs. Level-1 (observe) captures the accuracy of I.LMs to solve problems in existing benchmarks. These benchmarks may have been observed by the L.LSs in current similar form. Level-2 (mutate) captures the ability solve problem variations. Level-3 (imagine) captures the ability to correctly incorporate new logic into existing problems, even when this logic contradicts the original components. In certain cases, the new logic can be understood as counterfactual statement (Pearl, 2009).
Despite the introduction of various benchmark variations aimed at assessing LLMs' reasoning abilities, these varia- tions have been developed in an ad hoc manner, lacking a systematic hierarchy. Moreover, most existing approaches rely on significant manual effort and are designed for spe- eific tasks, making them difficult to scale across multiple benchmarks and tasks. For instance, in Mirzadeh et al. (2024); Srivastava et al. (2024), functional templates for simple math problems in the GSM8K benchmark (Cobbe et al., 2021) are manually created, restricting the analysis to only 100 new templates.
to evaluating reasoning in LLMs. As our experiments show, benchmarks across domains such as math, code, and logic can be systematically transformed using the same principles, generating challenging new scenarios that are unlikely to appear in the LLMs' pre-training data. Our findings indi- cate that all tested LLMs exhibit some degree of reliance statistical recall, while problems at higher levels in the reasoning hierarchy remain yet-unsolved challenge.
Contributions.
This paper presents three major contribu- tions, corresponding to each of the following sections:
In Section 2, we propose hierarchical framework to charac- terize existing and new approaches for evaluating reasoning in LLMs. The proposed hierarchy has three levels of increas- ingly difficulty that capture different levels of reasoning via variations of the problems of existing, well-established benchmarks.
Inspired by Judea Pearl and his ladder of causation (Pearl, 2009), our work expands on the traditional way to evalu- ate reasoning. Pearl asserts that: "Only machines that can correctly perform correlations, interventions and counter- factuals will have reasoning abilities comparable to humans." Following this principle, we present new framework, RE- IMAGINE, to characterize a hierarchy of reasoning abilities in LLMs, alongside an automated pipeline to guarantee scalable evaluations in each level of the hierarchy.
In Section 3, we propose an end-to-end, scalable pipeline that allows the generation of an arbitrary number of new problems in each level of the hierarchy. This is crucial to scale up current approaches that require the manual genera- tion of new scenarios.
RE-IMAGINE generalizes, expands and scales up evaluation of L.M reasoning by means of an pipeline with three core components:
In Sections 4-5, we use RE-IMAGINE to re-analyze the rea- soning abilities of all models in the GPT (Brown et al., 2020), Llama (Touvron et al., 2023), and Phi families (Kambhampati et al., 2024). We focus on four reasoning benchmarks: GSMSK for math (Cobbe et al., 2021), CLad- der for causality (Jin et al., 2023a), and CRUXEval (Gu et al., 2024) and Loop (Kamath et al., 2024) for code. We show consistent decline in LLM performance reasoning complexity increases across all evaluated benchmarks.
(i) A lanuuage-to-code model that converts each bench- mark question into a symbolic (code) representation. iii A set of mutations of the symbolic representation that creates an "executable" variation of the problem.
(iii) A code-to-language component that translates the gen- erated symbolic problems back into natural language.
The intermediate executable symbolic representation sures that correct outcomes can be calculated automatically from the mutations. This approach generates a diverse set of "unseen" " variations of existing, well-established bench- marks, providing novel challenges for LLMs.
Novelty. This paper introduces two major innovations:
The unified 3-level reasoning hierarchy presented in this paper incorporates both previously studied mutations and the newly introduced ones. According to this hierarchy, we highlight that prior research has mainly focused on Level-2 mutations, which assess a model's ability to generalize be-
RE-IMAGINE enables us to reinvent the standard approach
2

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Branpbiie Orggan Berchourk
Queetin to Syedbukk Represuratie
Musdice on Syededa Reprosentaion
Treedine oe tuuation
M
10 rodte
crto car becakans and bak cuffla you frmil aul nes, How wnet to
for tmailann and boun mallio with the Suutnee Jasel an lemger laker meattie
incaaddr sgg- www.bttttttttooo.... sgg- cafla, Agp
Iceaskke op kookioo.. Lpp -aalELL
Dacdd the Vatieed
[ANSWOR]
DACTATED ANSWEIU
Undrestad))
sgp- 58
ragie 33
HUG
Bouatee Ago
SEP to
mottts ogga on
rxttte op2- precee mours s8:2
aratf, sigin a pecc
rudle, uggn +3
bookfit
reata rpp
prke
reqaako up bookfat
illo prkc "oumadder
dii Fazz cumander
Saaek dadle
Acea'ss daika
Jawris dacle
Ske
Lirutyy
Jonet' dacla day
Jandts dedi
bruathet
She tree mecu lo
traittan bakes with
lemilfal bakoy will way-
rastie wil
bake with She wille
the wdl up
tho will rouatde
Ske
with
masikal: per troc
soatre renatake per per tid ago Bonk
Back fie maac
fack
dellaa
mil tiee beik dai How wach leare like Hiem wech toleo reakirs (io the make
inach tee masoo every day?
Levdle Olsene
Lead ot Mestaee
Levelge Innglee
The benehmark transformation pipeline (Section 3) outlined with an example from GSM8K (Cobbe et al., 2021). This pipeline leverages the symbolic representation of the question (a Python snippet form) automatically transform math QA problem (leftmost) into a similar format with additional reasoning steps (rightmoxt). Bottom: To clearly define the mutations, we transform the symbolic representation into computational graph (leftmost). Nodes represent variables from the symbolic representation, and edges illustrate the dependencies hetween them. The remaining six graphs depict the different types of mutations (see Section 3.2). E, B, M, R, P. and S represent variable eggs, breakfast.oggs. muffin_eggs. reminder, price, and sales, respectively. The yellow highlights indicate the modifications made relative to the original versions. Red nodes represent binary values, while all other nodes are mumerical.
Figure 1. Top:
yond existing benchmarks while maintaining the original reasoning path of the questions. In contrast, we empha- size that the Level-3 mutations introduced in our work are significantly more challenging.
has already seen training-set problems similar to the ones in the benchmark should be able to produce high accuracy.
Level-2 ("Mutate") captures the ability of LLMs to solve problems that have been mutated by, for example, adding irrelevant information, renaming values, changing val- ues. It tests the ability of models to generalize beyond the existing benchmarks in cases where the core logical re- quirements of the questions are preserved. Several works have proposed approaches that sit in this level (Mirzadeh et al., 2024; Srivastava et al., 2024; Wu et al., 2023; Lewis & Mitchell, 2024), which are based on manually created functional variations of the original problems. The results in such variations highlight memorization and fett-fitting issues. For a true reasoning model, the task performance should be invariant w.r.t. the class of changes in this level.
Alongside the reasoning hierarchy, we introduce to the best of our knowledge the first scalable mutation gen- eration pipeline that applies across multiple benchmarks and tasks. This framework enables the creation of an arbi- trary number of mutations at each level of the hierarchy for existing benchmark problems.
A detailed comparison with previous studies can be found in Table 4 in Appendix B.
2. RE-IMAGINE: The Ladder of Reasoning
Inspired by the ladder of causation (Pearl, 2009), we system- atically define a three-layer hierarchy ('observe', 'mutate", 'imagine") that characterizes different levels of reasoning abilities in L.Mss, in the same way that the ladder of causa- tion captures three different cognition skills. This allows us to characterize and compare the goals of different evalua- tion experiments with precision, both new and existing. A summary of the three levels is presented in Table !.!
Level-3 (<Imagine") is the topmost and most sophisti- cated level. It captures the models' ability to correctly incorporate new information and logic into existing prob- lems. Given a problem defined by a set of logical predi- cates or facts, this variation augments the original problem with an additional predicate that changes some previously stated one. Correctly incorporating new logic requires an accurate (explicit or implicit) representation of the steps required to solve the problem, as well the ability to contradict and revise prior knowledge. Counterfactual as- sessments (González& & Nori, 2024) sit on this level of the hierarchy. Task performance metrics and counterfactal related metrics like the probability of necessity (PN) and sufficiency (PS) can be used in this level as in (González.
Level-1 ("Observe") captures the accuracy (or other met- ric of interest) of LLMs on existing benchmarks. It is called observe because it is expected that an LLM which
We provide more detailed explanation of the fundamental connection between our hierarchy and causality Appendix C.
3

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
& Nori, 2024).
sentation and translated into NL. The remaining six columns in Figure 1 illustrate mutation variations.
Next, we detail how use the problems from benchmarks in Level-1 to create novel problems in Level-2 and Level-3.
Level-2 mutations
Sample Values assigns new values to all root nodes, When translating the mutation back to NL, only the values in the question are replaced with the new ones, while the rest of the narration remains unchanged. This mutation specifically aims to differentiate the model's 's reasoning ability from memorization caused by data contamination. We illustrate the pipeline with real example from the GSM8K benchmark, presented in the top row of Figure 1. Starting with a sample from the existing dataset, we first convert the question into executable symbolic form, c.g. code, knowledge graph (1) in Figure 1). In line with Tosh- niwal et al. (2024). we represent the math question as a Python code snippet. Next, we apply specific type of mu- tation to the symbolic representation (Q). In this example, we overwrite the value of the variable muffin_eggs eggs in the code. To maintain the core logic of the task a natural language-based math annestion-answering (QA) problem- we translate the change in the symbolic representation back into natural language (NL) and incorporate it into the origi- nal question (+). Due the executable nature of the Python code, the ground- truth answer for the mutated question can be obtained by running the modified code snippet (4).
3. Benchmark Synthesis Pipeline
We present a unified benchmark synthesis pipeline that auto- matically generates variations of existing benchmarks, pre- serving the core logic of the task while demanding stronger reasoning abilities to solve.
UselessInfo adds a new node dependent on a randomly selected node from the original graph, with the change de- scribed in NL between the context and the question. This introduces additional context, but does not alter original statements or impact the correct answer. This assesses the model's 's ability to disregard irrelevant information.
Level-3 mutations
AddDependence also introduces new node into the graph. However, unlike Uselessinfo, a randomly selected node from the original graph is modified to depend on the new node for its calculation. This is likely to influence the correct answer to the question. A natural way to encode this mutation in NL is to append a statement to the end of the original question, amending the original statement context, making this a Level-3 mutation.
Note that although Figure illustrates all the elements in the pipeline, not all of them will be necessary for every benchmark of interest. A summary of the required steps for the benchmarks discussed in this paper is provided in Table 2. Similarly, not all mutations are applicable to every problem, and additional mutations beyond those listed in Figure can also be considered.
Inser(Conditional adds a new node that connects two non-adjacent nodes in the graph, with edges linking the first node to the new node and the new node to the second node. In symbolic terms, this mutation is represented as an if-else condition. Two variables are randomly chosen, and one variable's value is set to 0 depending on the value of the other. Describing this in NI. as a change to the previous method of calculating the variable, it also becomes Level-3 mutation.
3.1. NL-to-Symbolii (i) & Symbolic-to-NL (})
Steps 1 and 3. corresponding to transformations from NL to Symbolic and Symbolic to NL respectively, are non- trivial. Benchmarks that have original problems already in code form (e.g. Loop and CRUXEval) do not require these steps. When they are required, they need some level of adaptation to the nature of the benchmark. Details for these two steps are provided in Section 4.
CounterFactual randomly selects a node in the graph and overwrites its value. Unlike Sample Values, this mutation does not directly replace the number in the question. In- stead, it presents the change as an assumption statement appended the original question. Thus, it modifies an existing statement in the context and adds an extra rea- soning step to the original question, making it a Level-3 mutation.
3.2. Symbolic Mutations ())
Bi-CounterFactual builds on CounterFactual to evaluate the model's s ability to connect the presence or absence of cause with its effect, an essential reasoning skill from the perspective of causation (Neuberg, 2003; Halpern & Pearl, 2005). Previous work (González & Nori, 2024; Hüyük et al., 2024) quantitatively evaluates this through necessity and sufficiency inconsistency rates (N-IR and S-IR), but relies on manually crafted questions and their counterfactuals. In contrast, our scalable pipeline unlocks
We showcase six code mutations spanning the Level-2 and Level-3 reasoning levels to create benchmark variations. To thoroughly define the mutations in @, we first convert the symbolic representation into a computational graph (left- most column in the bottom row of Figure 1). Nodes rep- resent variables in the symbolic representation and edges capture their dependencies. The mutation applied to the computational graph then reflected in the symbolic repre-

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Benchmark
Type
Input
Output
Required Steps
GSM8K
Mathematics QA
NL math question
Numerical answer
1
2
( ④
CLadder
Causal QA
NL causal query
Y/N answer
④
CRUXEval
Code understanding
Python function and input
Execution output
Loop
Loop invariant inference
C Code and an assertion
Y/N answer
Table 2. A summary of the steps used in GSM8K (Cobbe et al., 2021), CLadder (Jin et al., 2023a). CRUXEval (Gu et al., 2024), and Loop (Kamath al., 2024). Since the inputs for both CRUXEval and Loop are code snippets, steps ! and are not required.
large-scale analysis. In Bi- Hi-CoutterFactuaa,, the compu- tational graph is treated as Structural Causal Model (SCM), where the overwritten node acts as the cause and the final answer (leaf node) serves as the effect. Specifi- cally, Bi-CounterFactude requires binary cause and effect nodes, with the overwritten value ensuring a change in the cause statement's presence or absence.
integers are sampled from a discrete uniform distribution, while floats are drawn from uniform distribution centered around the original value. We also ensure that the final answer maintains the same type and sign as the original ground-truth answer.
Mutated Symbolic to NI 3 In the Sample Values mu- tation, only the values in the questions are replaced with newly sampled ones, while the rest of the narrative stays the same, so no new NL descriptions are required. For the other mutation types, we provide the original math question, its Python solution, and the code modifications to LLM (GPT- 4o), leveraging its text generation capabilities to describe the code changes in natural language. To guarantee the symbolic-to-No translation is correct, we prompt GPT-4o a second time to back-translate the mutated math problem into Python by modifying the original question's 's Python solution. The generated code must produce execution result that matches the ground truth answer of the mutated question. (Detailed prompts are shown in Figure 17 and Figure 16 in Appendix E.)
In the next two sections, we apply our automatic bench- mark synthesis pipeline to the math reasoning benchmarks GSM8K and CLadder (Section 4) and the code understand- ing benchmarks CRUXEval and Loop (Section 5).
4. Math Benchmarks: GSM8K and CLadder
This section first details the benchmark transformation pro- cess for GSM8K and evaluates its quality (Section 4.1). We then analyze model accuracy on numerical math ques- tions using the first five mutations outlined in Section 3.2, excluding Bi- CountterFactual (Section 4.2). Since Bi- CounterFactual involves binary questions and is primarily assessed with causation metrics, it is discussed separately in Section 4.3, Section 4.4 demonstrates that the findings from GSM8K extend to CLadder, another math reasoning benchmark focused on probabilities and causality.
To verify the accuracy of the mutated QA pairs, we man- ually reviewed 50 randomly selected examples from each mutation type. Valid examples are the ones that contain a clearly defined question and a correct ground- an- swer. The percentage of invalid QA pairs in Sample Values, UselessInfo, CounterFactual, AddDependence, and Insert- Conditional were 3.33%, 0.00%, 6.67%, 5.00%, and 5.00%, respectively.
4.1. Transformation Pipeline
Question to Symbolic Representation ! Toshniwal et al. (2024) introduced OpenMathInstruuct, whose valida- tion set contains 970 GSM8K QA examples paired with Python solutions generated by Mixtral-8x7B (Jiang et al., 2024). We construct our test set by filtering out examples where the Python solution execution does not match the ground-trutte truth answers. To ensure high-quality mutations, we further filter the data to keep only those where all constant variables in the code (root nodes in the computational graph) align with the numbers in the question and vice versa. 2
4.2. Reasoning on Numerical Math QA
In line with previous studies, during testing, all models are provided with 8 in-context examples with Chain-of-Thought (Co'T) to help them understand the task (prompt shown in Figure 18 in Appendix E).² We evaluate models from three popular families-PPl Llama, and GPT (see Table 3 in Appendix A for details).
Symbolic Representation to Mutation 2
We incorpo- rate all six types of mutation described in Section 3.2. Since most GSM8K questions are framed within a story context, we ensure that newly sampled values for existing variables align with the original value's type (float/integer) and sign to preserve the story's 's coherence. Within this constraint,
The answer accuracies are presented in Figure 2, with the percentage of invalid mutated examples in each mutation category displayed as a hashed block above each bar. By assuming that models would correctly answer these ques- tions if the QA pairs were valid, this estimation serves as an
We account for commonsense numerical facts, such as one year having 12 months, and one hour containing 60 minutes.
We follow this blog post to strengthen our prompt.
5

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
95%
Sampletalues
85%
Uselessinfo
80%
75%
70%
CounterFactual
InsetCoodttional
65%
AddDependence
60%
1222
Raw Emnr
55%
50%
25%
20%
15%
10%
lamaa
Ilama3.1
Liama3-708
GPT-3.5
4PE4
GPT-40
GPT-o1 (+1)8))
phiB-mini
phi3-small phia.5-mini
phiB-medium phi4
OBL
17081
(11735)
(9381
(n)) (148)
(148)
Figure 2. GSM8K results summary: model accuracy on numerical answer predictions across test set variations in different reasoning levels
(see Section 3.2 and Figure 1). Due to polential noise from the automatic mutation process, we performed a human evaluation on the mutated test set. We added the percentage of invalid examples in each mutation category to the top of each accuracy bar as hashed blocks, assuming that these QA pairs were correct, the models would answer them correctly. This provides an upper bound on the model's s performance. We also present box plot (see Figure 10 in Appendix E) to illustrate the statistical accuracy across 10 scts of samples.
Modelss S-IR and N-IR on binarized GSMBK (Zero-shot)
upper bound on the model performance.
0.5
Senias
Sumtiency incondistncy [s
GPALL
Key findings are: (1) With 8-shot in-context examples, most of the models achieve high accuracy (~ 95%) on the Level- 1 raw test set. (2) Among Level-2 mutations, UselessInfo is less wwaieeniinnn-speccinee
phics Setrd
0.3
prchimal
for larger models indicatnn their ability to ignore irrelevant details. However, nearly all models experience around 10% accuracy drop on Sample- Values, despite unchanged reasoning paths and only altered values. (3) Level-3 mutations pose greater challenge, with models showing significantly lower upper-bound perfor- mance than on Level-/ and 2 test sets.
0.2
moweiittimaaaa 708
oPT-1
0.00
0.05
0.10
0.15
0.20
Necessity Inconsistenyy (K-RR)
Figure 3. Sufficiency/necessityy inconsistency rates (S- (S-IR/N-IR) on GSM8K factuul//cuntteffactaall test set. Models located near the bottom-left corner are thought to predict the causal relationship between the cause and effect, i.e. sufficient and necessary, in a way that is consistent with the true causal relationship, as defined by the ground truth.
We also conduct ablation experiments to examine per- formance on test set containing multiple mutations (Ap- pendix E.3). We found that composing mutations increases performance gap between the mutated and the original sets.
and sufficiency inconsistency rates (N-IR, S-IR) introduced in Hüyük et al. (2024), which account for the errors in the approximation of these measures in a normalized way (an optimal reasoning LLMs is one with N-IR = 0 and S-IR 0). Because obtaining N-IR and S-IR is computationally expensive, we used 50 questions from the validation set of the benchmark, where the condition node
4.3. Reasoning Evaluation with Binary Counterfactuals
Bi- Ci-CoutterFatual as described in Section 3.2 creates two auxiliary nodes in the computation graph with binary ver- sions of a condition and an outcome. The reason for consid- ering this problem transformation is that it allow us to com- pute metrics that are relevant to evaluate reasoning beyond accuracy. As shown in González & Nori (2024); Hüyüik et al. (2024), this scenario enables the computation of the probabilities of necessity (PN) and sufficiency (PS) from the counterfactual literature (Pearl et al., 2000). Intuitively, these measures capture the probability of activating/deacti- vating a binary outcome in the presence/absence of a binary input. Although this restricts our analysis to the simplified (binarized) version of the GSM8k introduced by the Binary Counterfactual mutations, we compute these metries due to their intrinsic value.
randomly sampled across the available leaf nodes. To obtain these results, we follow the same setup as in Hüyük et al. (2024).
Figure 3 shows the average S-IR and N-IR for all models across 50 random examples. Consistent with the numerical accuracy evaluation, GPT-ol remains the hest-performing model, while Llama 8B models, GPT-3.5, and GPT-4 are at the other end of the spectrum. However, GPT-4o and Llama 70B outperform phi3-small and phi3-mini in the causal reasoning evaluation.
4.4. Similar Findings on CLadder
To confirm that our findings in GSM8K generalize to other math reasoning benchmarks, we use the transformation pipeline (Section 3) to automatically generate three test set variations for CLadder, causal reasoning benchmark.
The ground truth PN and PS varies across problems and nodes. To give a benchmark-level measure of how well different models approximate them, we use the necessity
6

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Rlaalder Original Esample)
dell Jicweet
[Oructvak Origiaa Ecarppi)
Por brudundd stan con't see the aoom and siieee that doe't adl skem, (be prduuislay sttuess blam a the
Soe
55%
Klars, the pootability of ringing alaca lopels Foc. hadoodal I For bodlards that ake in the
hraduasde
damn
007%
oucpec
50%
Redefine Function (L2)
Ansers 31 7111
Replace Operator
effict clans chock
recilatior effeet thoough wife, would Lumibrra peollvely
70%
75%
oatpat
Diratter Netachaba
40%
Snap Conditiona
Hotate Value 0.31
Nutate String (L3)
(Ctadder CounterFactral)
ezepu apperd reces cow.t 201, 304
Redetine Function (L.)
and sores IVs. the hedoondttion don't os the zomn and vilaw thve set the daen I.) Junzen the mnckahiit of the clame piopieg,
brabands Suxt pro rhiging
05%
ectues witte
30%
Swap Conditional (L33
Replace Openator (L.3
607
25%
aat Eouam))
Thrrge WeN Worele diliing agnn The teviiitton Filce Bncegh vas wost triliadd piali slame clock?
30%
for i
20%
50%
15%
Aapert 3,3.3233,,2221 73
Figure 4. Left: An example from CLadder test set and its Counter- Factual mutation. Right: CLadder accuracy of the best-performing model in each family set variations at different reasoning levels. The full results can be found in Figure 21 in Appendix F. Undelatedtrreeeevnnttfo and RelatedIreleevantfoo are categorized as Uselessinfo per Figure 1.
up
5%
0%
causal question answering across test
GPT-o1
Liama 3.1 (88) phi3-small
Figure 5. Left: An example from CRUXEval dataset and the Level- and Level-3 versions of its Mutate mutation. Right: CRUX- Eval accuracy of the best-performing model in each family. Full results can be found in Figure of Appendix G.
We examine three mutations UnReeatedIrrelevvnnnffo Re- latedIrreleannInfo, and CounterFuctuall, with the first two classified under UselessInfo (see Figure and Section 3.2). Details on the benchmark, implementation, LLM prompts, and mutation specifics are provided in Appendix F. The accuracy of the best- pertf-eerorming model in each family is shown in Figure 4. Similar to our observations in GSM8K, Level-3 mutations present a greater challenge than Level-2, causing models to exhibit an approximately 20% drop in accuracy relative to the original test set. Models exhibit lower accuracy on test set variations composed of two types of mutations (Figure 21 in Appendix F).
on the task if Frama- C (Correnson et al.) verifies via SMT solvers (de Moura & Björner, 2008) that the LLM output is loop invariant, and fails otherwise (details in Appendix H).
Since both tasks are already programs, steps ! and ② of the pipeline are not required.
5.1. Symbolic Mutations to CRUXEval ②
Each CRUXEval mutation can be applied either directly to the function code (Level-2), or presented as a code diff in an "imagine" statement following the original code (Level- 3). The Level-2 versions, in particular, are designed to have minimal effect on code length, execution time, and overall question difficulty. Table 6 in Appendix G shows the different types of mutations implemented for CRUXEval, summarized below.
5. Code Benchmarks: CRUXEval and Loop
We investigate two code understanding tasks: input/output prediction (CRUXEval) and automatic inference of loop invariants (Loop).
Replace Operator replaces an operator of type ast.BinOp, ast UnaryOp, BsolBo or ast AugAssign with its inverse or negation, where applicable.
The CRUXEval benchmark (Gu et al., 2024) consists of 800 short LLM-generated Python functions alongside input- output pair for each function (see Figure 5 for example). The model's task is to predict the output of the function when evaluated on a given input parameter. Functions are
Mutate String replaces a string instance with a uniformly random character sequence of the same length.
filtered to include only those with low computation and
memory requirements, with no side-effects or randomiza- tion, and excluded arithmetic calculation.
Mutate Value changes the value of an instance of type bool, int, or flaat. Booleans are replaced with their negations: integers floats are perturbed by a uniformly random nonzero integer / float (respectively) in [-10, 10).
The Loop benchmark (Kamath et al., 2024) contains loop invariant inference tasks, each of which consists of pro- gram with loop and an assertion (see Figure 6 for example). The goal is to infer a predicate that satisfies the following three conditions: it holds before the loop starts executing, holds for each iteration of the loop, and implies the assertion when the loop exits." 4 The model succeeds
Swap Conditional selects a random conditional node for modification. If both an if and e else branch are present, the code body for each branch is swapped. If only an if branch is present, the condition of the branch negated.
Redefine Function defines a wrapper for a random non- attribute function, and replaces call to the original func- tion with call to the wrapper.
AAthoouhh inferring such loop invariants is undecidable, in practice, checking whether given candidate invariant satisfies the three conditions can he done well by automated software verifica- tion tools like Frama-C (Correnson et al.).
A subsequent validation pass permits only code which ter- minates and returns a value within 5 seconds without er-
7

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Hoop: Original Esample]
invariant of the original program is also a valid loop invariant for the mutated program.
naint) 1.
85%
85%
Pee pnompriee veteorrgnno
while iy 1000000
75%
70%
//baxaurt (o0ley))
60%
55%
5.3. Model Performance
50%
[Mutation: Head Original (Levekaa))
41%
Zero-shot accuracy on CRUXEval tasks are summarized in Figure 5. Overall, accuracy scores decrease for both Level-2 and Level-3 mutations compared with factual prob- lems, with Level-3 posing the greatest challenge. The drop in performance on Level-2 mutations is particularly strong indication of memorization effects from benchmark leakage, as these mutations intentionally introduce minimal change to the execution complexity and corresponding code under- standing skills. Notably, the original CRUXEval benchmark is publicly available and I.I. --ggeeerrrtedd both of which provide potential leakage pathways.
ind maintl cot 2. y al-- 2.
15%
30%
10c0001
25%
13%-
20%
Figure 6. Left: An example from Loop dataset and its Read Orig- inal mutation. Right: Loop concise results- accuracy of the best-performing model in each family. Full results can be found in Figure 25 of Appendix H.
rors. Code transformations for CRUXEval are deterministic, since for any validated program, the output of the code on the provided inputs is taken to be ground truth. See Ap- pendix G for additional implementation detail.
Evaluations on Loop tasks are summarized in Figure 6 (for evaluation with all models, see Figure 25 in Appendix H). Smaller models show performance degradation on introduc- ing junk named variables. The performance decline from "Junk Hinr" to "Junk No-Hint" shows that LLMs use vari- able names as semantic cues. The "X-Original" mutations confuse all models and significantly degrades their success rates compared the original tasks.
5.2. Symbolic Mutations to Loop
Unlike earlier benchmarks, applying substantial mutations to Loop tasks is more challenging. After changing the values of the program variables, loop invariants can cease to exist. Hence, we limit ourselves to Level-2 mutations that add irrelevant information in the form of additional variables and operations, leaving the values of variables from the original program unaffected. Table 7 of Appendix H describes these mutations, summarized below.
6. Ablation Study and Discussion
We further evaluate the models' reasoning abilities through two ablation experiments: (1) to determine whether the performance drop on mutated questions stems from the mutations themselves or from the additional reasoning com- plexity introduced by some mutations; and (2) to assess the impact of in-context learning examples on the models' reasoning performance.
Junk Hint adds five new variables junk_0, junk_1, etc., to the program (Si et al., 2018). Before the loop, they are initialized with randomly generated constants. Within the loop body, each new variable is updated with arithmetic expressions over randomly generated constants and the new variables.
Junk No-Hint assigns names to the new variables that resemble those in the original program. This aims to prevent LLMS from identifying variables with junk in their name as unnecessary.
6.1. The Influence of Reasoning Complexity
We use GSM8K as testbed for our analysis. Following Ye et al. (2025), we quantify the complexity of a numerical rea- soning question by the number of reasoning steps, defined as the number of operations in its code solution. By this measure, all mutated questions introduce one additional rea- soning step compared to their original counterparts, except for those in the Sample Values category. Notably, while Use- lessInfo questions also include an extra reasoning step. this step does not affect the final answer, so we still categorize UselessInfo as a Level-2 mutation. Using this definition, we compute the average accuracy of each model across exam- ples with different numbers of reasoning steps. Given the large number of models tested, aggregate their results
Read Original reads the original variables of the code into the newly introduced ones. Original variables and op- erators randomly sampled and added to new variables.
Write Original introduces superficial additional writes to the original variables, i.e. they leave the values taken by these variables at runtime unaffected. For a new variable y, existing variables are incremented by an identically zero polynomial fly). To further obscure the underly- ing identity, polynomial coefficients are decomposed and rearranged.
X-Original applies both Read and Write Original.
'Each mutation type applies only to subset of the questions in CRUXEval, so we also evaluate performance on each mutation with respect to the corresponding (factual) subset of problems (see Figure 24 in Appendix G).
Although the mutated programs are syntactically larger, all mutations described above have the property that the loop
8

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Average Astunicy ve Reeseoning Difiouly stroses All Texted Models
Summary of the models' performance with different In-context examples
200%
25.0
Fan Only
30%
Now
223
firee
ama3-
20.0
Standard deviation ACC
60%
19.2-
X
*x sasi
17%
22.3
B-mmollum
00.0
Reascring Dificuily jas the number resansing staps)
Figure 7. We analyze model performance across varying numbers of reasoning steps in the GSM8K benchmark. The x-axis rep- resents the number of reasoning steps, defined as the number of operations in the code solution. The plot aggregates the perfor- mance of all tested models. Detailed results for each model are provided in Figure 15 in Appendix F.5.
Average ACC (%)
Figure 8. Models' performance when prompted with different types of in-context examples. We summarize model performance across two dimensions: the average accuracy across all mutated test sets and the original test set (x-axis), and the standard devi- ation in accuracy on the five mutated test sets w.r.t. the raw test set (y-axis). The bottom-right corner reflects the ideal balance of high performance on both raw and mutated sets. In this plot, dots indicate models prompted only with original examples, crosses represent models prompted only with mutated examples, and trian- gles denote models prompted with both original and corresponding mutated examples. Model structures are differentiated by color, with the same model represented by the same color.
and report the overall average accuracy in Figure 7. Detailed per-model results are in Figure 15 in Appendix E.5.
From Figure 7, we draw two key observations: (1) even when controlling for the number of reasoning steps, mod- els consistently perform worse on mutated questions than on the original test set; and (2) Level-3 mutations pose a substantially greater challenge than Level-2. Notably, the accuracy on Level-3 mutations with just three reasoning steps is significantly lower than on original test examples requiring seven reasoning steps. These findings suggest that the primary cause of performance degradation on mutated questions is the mutations themselves, rather than the added reasoning complexity.
find that the overall performance of all LLMs degrades with increasing ladder levels. The ol model shows greater robust- ness on this benchmark and achieves the best performance among all tested models on bi-counterfactuals. However, even the most powerful models struggle with structured problem variations, particularly when these variations are combined. Similar results were observed on CLadder.
Additionally, we successfully applied RE- to mu- tate code benchmarks Loop and CruxEval. Despite the mutations being designed to have minimal effect code length, execution time, and overall question difficulty, a substantial decline in performance is again observed across models and problem variations. This lends additional evi- dence that these models struggle with higher levels of the reasoning hierarchy, suggesting that gaps in their capabil- ities may be hidden by effects such as memorization and benchmark leakage.
6.2. The impact of in-context examples
Following previous studies, the 8-shot in-context examples used in the GSMSK experiments in Section 4 are randomly sampled from the original training set. Here, we further investigate whether including mutated examples in the in- context examples can improve model performance on the generated test set variations. As shown in Figure 8, we find that most models achieve significantly higher accuracy on the test variations when given both original and mutated examples, compared to using only original GSM8K exam- ples or only mutated examples as in-context examples. A detailed explanation of the experiment, including additional results and full prompt examples, is in Appendix E.2.
In addition to the models discussed in previous sections, we report results for other popular models in Appendix D. The conclusions are found to generalize well to these models.
RE- IMAAINE provides a systematic framework to assess and expose these weaknesses, highlighting the need for more rigorous evaluation strategies. Expanding RE-IMAGINE to broader domains could further enhance our understanding of reasoning capabilitics and limitations of LLMs. Our hope is for RE-IMAGINE to provide a foundational framework for a more nuanced evaluation of LLMs and encourage the development of more robust reasoning models.
7. Conclusion
This work presents a novel framework, RE- RE-INAE de- signed to assess the reasoning capabilities of LLMS through the systematic generation of challenging problem variations. Our findings reveal a consistent decline in model perfor- mance as reasoning complexity increases across all evalu- ated benchmarks. On our mutated GSM8K benchmark, we

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Impact Statement
Cobbe, K., Kosaraju, V., Hilton, J., and Schulman, J. Train- ing verifiers to solve math word problems. arXiv preprint arXiv:2110. 2021.
The RE-IMAGINE framework represents significant ad- vancement in evaluating the reasoning capabilities of Large Language Models. By systematically generating problem variations that cannot be solved through memorization alone, RE-IMAGINE provides a robust method to disentangle gen- uine reasoning from statistical recall. The findings reveal notable drops in LLM performance when faced with prob- lem variations. By setting new standard for evaluating hierarchical reasoning, this research paves the way for more transparent and interpretable AI systems, ultimately con- tributing to the broader goal of building models with true generalizable reasoning capabilities. This will impact the development of future AI systems. It will also influence benchmark design, ensuring that evaluations better reflect genuine cognitive capabilities rather than memorization. Ad- ditionally, this research can guide policymakers, researchers, and practitioners in designing more reliable and trustworthy Al applications across domains such as education, health- care, and scientific discovery, where robust reasoning is critical.
Correnson, I.., Cuoq, P., Kirchner, F., Maroneze, A., Pre- vosto, V., Puccetti, A., Signoles, J., and Yakobowski, B. Frama-C User Manual. URL http: //rt::////raaaaa/ com/ dwwp/o/d///l//o/oco/oc/ccc...o// frama- C- user wwwsser-mantnn manuall pdf.
de Moura, L. and Bjørner, N. Z3: An efficient smt solver. In Ramakrishnan, C. R. and Rehof, (eds.). Tools and Algorithms for the Construction and Analysis of Systems, pp. 337-340, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg. ISBN 978-3-540- 74800080
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. Pal: Program-aided lan- guage models. In International Conference on Machine Learning, pp. (0764-1079) PMLR, 2023.
Glazer, E., Erdil, E., Besiroglu, T., Chicharro, D., Chen, E., Gunning, A., Olsson, C. F., Denain, J.-S., Ho, A., de Oliveira Santos, E.,, Järviniemi, O., Barnett, M., San- dler, R., Vrzala, M., Sevilla, J., Ren, Q. Pratt, E., Levine, L., Barkley, G., Stewart, N., Grechuk, B., Grechuk, T., Enugandla, S. V., and Wildon, M. Frontiermath: bench- mark for evaluating advanced mathematical reasoning in AI, 2024. URL https: //aaxvvvvvv////////////////aaav. wwtpxiiivoor/a/aeeee www./bss////..o...o..........s.s 2411 04872.
References
Ait EI Hara, H. R., Bury, G., and de Oliveira, S. Alt- Ergo-Fuzz: A fuzzer for the Alt-Ergo SMT solver. In Keller, C. and Bourke, T. (eds.). Journées Franco- phones des Langages Applicatifs, pp. 235-244, Saint- Médard-d' Excideuil, France, June 2022. URL https: //inria hal.hal.l.. wwc.enceeaaale... 003 62033666662 6861.
González, J. and Nori, A. V. Does reasoning emerge? ex- amining the probabilities of causation in large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Barrett, C. W., Conway, C. L.,, Deters, M., Hadarean, L.., Jovanovic, D., King, T., Reynolds, A., and Tinelli, C. CVC4. In Gopalakrishnan, G. and Qadeer, S. (eds.), Computer Aided Verification 23rd International Confer- ence, CAV 2011, I, Snowbird, UT; USA, July 14-20, 2011. Proceedings, volume 6806 of Lecture Notes in Computer Science, pp. 171-177. Springer, 2011. doi: 10.1007/ 978- 778-3-642-22210--1)..1 URL https://p//c/ http:/doi.oo /oo..ogg/////oooooooo/ooooooooo. 000-998-----2-2--------11
Gu, A., Rozière, B., Leather, H.,, Solar-Lezama, A., Syn- naeve, G., and Wang, S. I. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.0306 2024.
Halpern, J. Y. and Pearl, J. Causes and explanations: A structural-mode approach. Part I: Causes. The British journal for the Philosophy of Science, 2005.
Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint ar XxXv:2000.033000, 2020.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. arXiv preprint 2005:20005.4166 2020.
Holyoak, K. J. and Morrison, R. G. (eds.). The Cam- bridge Handbook of Thinking and Reasoning. Cambridge University Press, Cambridge. England, 2005. ISBN 9780521824170
Chollet, F., Knoop, M., Kamradt, G., and Landers, B. Are prize 2024: Technical report. arXiv preprint arXiv:2412.04604, 2024.
Huang. J. and Chang, K. C.-C. Towards reasoning in large language models: A survey. In Rogers, A., Boyd-Graber, J., and Okazaki. N. (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 1049-1066 Toronto, Canada, jul 2023. Association for Computational Linguistics.
Chollet, F., Knoop, M., Kamradt, G., and Landers, B. Are prize 2024: Technical report, 2025. URL https:// arxiv. or.o abtpaa/s..22/./////// 04604.
10

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Hüyük, A., Xu, X., Maasch, J., Nori, A. V., and González, J. Reasoning elicitation in language models via counter- factual feedback, 2024. URL https: /ttp://aaxvvvvoooo org/ abs/ 2410/221... 03767.
OpenAI. Early access for safety testing, Decem- ber 2024.
https://openai.co com/ index/ early-access- acceaaccccsseeee sar-sss-----a-eeeeeeee early-
Pearl, J. Causality: Models, Reasoning, and Inference. Cambridge University Press, Cambridge, England, 2nd edition, 2009. ISBN 1996666..
Jaech, A., Kalai, A., Lever, A., et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. I., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint ar Xiv:2401.040880 2024.
Pearl, J. et al. Models, reasoning and inference. Cambridge, UK: CambridgeUniivrrityyyen
19(2):3, 2000.
Shi, F. Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., Schärli, N., and Zhou, D. Large language models can be easily distracted by irrelevant context. In Inter- national Conference on Machine Learning, pp. 31210- 31227. PMLR, 2023.
Jin, Z.,, Chen, Y., Leeb, F., Gresele, L.,, Kamal, O., Lyu, Z., Blin, K., Gonzalez, F., Kleinen-Weiner, M., Sachan, M., and Schölkopf, B. CLadder: Assessing causal reasoning in language models. In NeurlPS, 2023a. URL https: // //oppnreeiee.../ net. for:m 2== JxOYqu.
Si, X., Dai, H., Raghothaman, M., Naik, M., and Song, L. Learning loop invariants for program verification. In NeurlPS 2018, 2018.
Jin, Z., Chen, Y., Leeb, F., Gresele, L., Kamal, O., Lyu, Z., Blin, K., Gonzalez Adauto, F., Kleiman- Weiner, M., Sachan, M., et al. Cladder: Assessing causal reasoning in language models. Advances in Neural Information Processing Systems, 05310088-1066 2023b.
Srivastava, A., Wei, J., Jun, H., et al. Beyond the imita- tion game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.0444 04.15, 2022.
Kamath, A., Mohammed, N., Senthilnathan, A., Chakraborty, S., Deligiannis, P., Lahiri, S. K., Lal, A., Rastogi, A., Roy, S., and Sharma, R. Leveraging Ilms for program verification. In FMCAD, 2024. URL http: //hd1, handle e. net 500. 1270022000785
Srivastava, S., PV, A., Menon, S., Sukumar, A., Philipose, A., Prince, S., Thomas, S., et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint arXiv:2402. 19450, 2024.
Team, D. A. Deepscek-rl: A comprehensive reasoning model. DeepSeek Al Research, 2025. URL https: //qithub. cwm/ wwwpseee-eiieeeeeeeeeeeeeeeeeeee aww.aaeeeeeeeeeeeeeeeeeeee -R. Available on GitHub.
Kambhampati, R. S., Iwafuchi, T., and Dasaka, Phi-3:
A family of small open models. Microsoft Al Research, 2024. Pre-release information.
Lewis, M. and Mitchell, M. Evaluating the robustness of analogical reasoning in large language models. arXiv preprint arXiv:2411.14211 2024.
Toshniwal, S., Moshkov, I., Narenthiran, S., Gitman, D., Jia, F., and Gitman, I. Openmathinstruct-il A 1.8 million math instruction tuning dataset. arXiv preprint arXiv: Arxiv-2402. 10176, 2024.
Li, Z., Zhou, Z., Yao, Y., Li, Y.-F. Cao, C., Yang, F., Zhang, X., and Ma, X. Neuro-symbolic data generation for math reasoning, 2024. URL httptt / //txxvxvvvv///////////////////// v. org/ abs/ 2412. 04857.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Roziere, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
Mirzadeh, I., Alizadeh, K., Shahrokhi,, H., Tuzel, O., Bengio, S., and Farajtabar, M. Gsm-symbolic:: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05224 2024.
Valmeekam, K., Olmo, A., Sreedharan, S., and Kambham- pati, S. Large language models still can't 't plan (a bench- mark for Ilms on planning and reasoning about change). NeurIPS 2022 Foundation Models for Decision Mak- ing Workshop. 2022.
Mitchell, M. and Krakauer, D. C. The debate over under- standing in ai's large language models. Proceedings of the National Academy of Sciences, 120(13):1-15, 2023.
doi: 10. 12..173/paa..2215571122222 2212222222
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Super- glue: stickier benchmark for general-purpose language understanding systems. arXiv preprint 100507790,.00537, 2019.
Neuberg, L. G. Causality: models, reasoning, and infer- ence, by judea pearl, cambridge university press, 2000. Econometric Theory, 194))675-688, 2003.

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Webb, T., Mondal, S. S., and Momennejad, I. Improving planning with large language models: A modular agentic architecture. arXiv preprint arXiv:2310.0013: 2024.
Weidinger, L.,, Mellor, J., Rauh, M., al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2 12.04399, 2021.
Wu, H., Barrett, C. W., and Narodytska, N. Lemur: In- tegrating large language models in automated program verification. ICLR, 2024.
Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., and Kim, Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint axXiv22307.02477 2023.
Xu, F., Hao, Q. Zong, Z., Wang, J., Zhang, Y., Wang. J., Lan, X., Gong. J., Ouyang, T., Meng, F. al. Towards large reasoning models: survey on scaling Ilm reason- ing capabilities. arXiv preprint arXiv:2501, 09686, 2025.
Yang, L.., Shirvaikar, V., Clivio, O., and Falck, F. A critical review of causal reasoning benchmarks for large language models. In AAAI 2024 Workshop on' "Are Large Language Models Simply Causal Parrots?". 2024.
Ye, T., Xu, Z., Li, Y., and Allen-Zhu, z. Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process. Proceedings of the 13th International Conference on Learning Representa- fions, ICLR '25, April 2025. Full version available at https: / ssrn. com/ abstract=
Zhang. Z., Chen, J., and Yang. Darg: Dynamic evaluation of large language models via adaptive reasoning graph, 2024.
Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., and Han, J. Don't make your I1 an evaluation benchmark cheater. arXiv preprint arXiv:2311.. 01944444 2023.
Zhu, K., Chen, J., Wang. J., Gong, N. Z., Yang, D., and Xie, X. Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference Learning Representations, 2023.
12

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
A. Appendix: Language Model Details
Models
Details
#Parameters
Pre-train Size (tokens)
Phi-3 mini
microsoof/Phii--mmii- 128k-instruct
3.8B
4.9T
Phi-3.5 mini
miccrosof/Phi-335--iii---- inntiiicttttctt
3.8B
3,4T
Phi-3 small
microssot/Phi---smalll- 1ammalll-28--msstrrcc
7B
4.8T
Phi-3 medium
microxaft/hii---meddun 128k-instruct microsoft/phi-4
14B
4.8T
Phi-4
14B
9.8T
Llama
meta Saata-llam/LLiama22 2-7-chhtt-t--t--
7B
2T
Llama 3
meta- Patte-llmma/MMta-lLama--s -a-------amm-------------
8B
15T
Ilama 3.1
Patte-llama/MMtta-Aaaaaaaa -tttttttuuut
8B
15T
Llama 3.3 (70B)
meta-llama//Meta- -aallommnmMMaa---mmm-....33333. J.mm--.3--70---- I.tt-turtttttt
8B
1ST
Llama (70B)
meta-llamalee Satalllaaa/Mett-aaa -aattta--aaaa----0000 3-no----------ee 70B-Instruct
70B
15T
GPT-3.5
8pt-35-turbo 1106
175B
GPT-4
gpt-4-32k 0613
GPT-4o
gpt-4o 2024-08-06
GPT-ol
ol ol-preview 2pppviiew2022-09-11 2024-09-12
Table 3. Details of I.IMs used in this work.
B. Appendix: Comparison to Related Work
Previous Studies
Mutations
Manual Effort
Scalability
GSM-IC (Shi et al., 2023)
UselessInfo (I.2)
Hand-written patternsrrulss for each question
Scaling within questions
iGSM
Out of scope
Human--efined concepts, dependency graphs, NL. expression patterns for QA pairs.
Scaling within questions
(Yc et al., 2025)
constructed from scratch
GSM-Symbolic (Mirzadeh et al., 2024)
UselessInfo (L2)
wiritenitten patterns/rules for cach question
Scaling within questions
SampleValues (L2)
Changing Names (L2)
-Srd (Gao al., 2023)
SampleValucs (L2)
Writing prompts get symbolic representation. Writing program to sample values for parameters.
Scaling within benchmarks, since the code generation replies ground truth CoT.
CounterFactual (Höyük ct al., 2024)
Bi-Counterfactual (L3)
Hland-writted patterns/rules for each question
Scaling within questions
Re-Imagine (Ours)
Covering L2 and L3
Customizing reusable adapters
Scaling across benchmarks
Table 4. Compared existing work. Scaling within questions means that for each original question with human mwrtann patterns, an arbitrary number mutated questions can generated. Scaling within benchmarks means that mutation generation tool could help generate arbitrary number of mutated questions for all questions in the entire benchmark.
C. Appendix: Fundamental Link to Causality
The three levels of Pearl's ladder of causation intervention, correlation and counterfactuals provide a foundational hierarchical characterization for the reasoning levels of an AI system. Our work inspired by this ladder, and taking the three levels reference, we describe a hierarchical of experiments that also capture different levels of reasoning in LLMs. The key connection between both frameworks the problem's computation graph, which can be understood as causal model in Pearl's framework. Each problem in benchmark can be interpreted specific node values. Experiments associated with different perturbations in such graph can often be related to operations in Pearl's ladder of causation. For instance, computing the effect of change to one leaf node maps to the standard definition of a counterfactual from causal inference. Note, however, that not all mutations the three levels have a causal counterpart (for example, adding an irrelevant piece of information or changing an operation). In this sense, our framework covers a broader scope of reasoning tasks each level.
single realization of the graph with
13

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
D. Appendix: Extra Models Performance on GSM8K, CruxEval and Loop
Extrall Model Acouricies on GSMAK Original In-corient Examples]
Extra Model Accuracies CruxEval
Extra Model Accuracies Loop
15%
50%
Taw
25%
55%
Made Meae (a)
70%
basnConditional
3mh-
AccDeponcence
45%
55%
00%
25%
50%
60%
50%
am-
20%
20%
255
20%
25%
25
20%
15%
2%
10%
NWAC 322
GPT45
0%
0Wb U2M
Figure 9. In addition to the models discussed in the main body of the paper, we also report results for several popular models on GSM8K, CruxEval, and Loop. The conclusions drawn the main paper are found to generalize well to these additional models.
E. Appendix: GSM8K Details
E.1. Statistical Accuracy on GSM8K and its variations
Statistical accuracy of Models over sets of samples
80
Accuracy (%)
9
40
Samplevalues
Uselessinfo
CounterFactual
InsertConditional
AddDependence
llama2
Ilama3
llama3.1
Llama3-708
GPT-3.5
GPT-4
GPT-4o
GPT-o1
phi3-mini phi3. 5-mini phi3-small ph3-mmdium
phi4
Models
Figure 10. The statistical accuracy of models on GSMSK numerical answer predictions is evaluated across different mutated test sets at varying reasoning levels (see Section 3.2 Figure 1). generate this plot, we randomly select examples eligible for all mutations, excluding Bi- Bi-CounterFactuud We then create 10 test variations for each mutation by sampling with 10 different seeds. All models are independently tested on these 10 variations, and the resulting accuracics are used to generate the bar plot. During ohain-of-Thhught testing, each model is prompted with 8 raw examples along with their Chain-of- Thought (CoT) process. Our observations are follows: (1) All models show significant accuracy variability when tested on mutated test sets, particularly on variations with Level-3 mutations; (2) For all models, the average performance the mutated sets is notably lower than the original GSMSK test (as indicated the dashed line). Moreover, for most test set variations with Level-3 3 mutations, even the accuracy on the most beneficial sample set falls helow the accuracy of the original GSM8K test. This suggests that single sentence mutation altering the computational logic of the original math question may severely impact the models' performance.
pool of examples
E.2. The Influence of In-context Examples
In this section, we examine the impact of in-context examples on the model's reasoning ability.
First, we aim to investigate whether providing the model with only mutated examples can improve its reasoning performance on the generated test set variations. During testing, all models are provided with seven mutated examples with amended CoT process, each from a different type of mutation (including two additional mutations beyond those discussed in Section 3.2 and Figure 1). Figure 19 shows one detailed prompt. Figure presents the results, indicating that the models exhibit
14

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
behavior highly similar when original GSM8K examples are used in-context examples (see Figure 2).
Next, we investigate whether providing the model with both the original example and its mutated variations can enhance its reasoning performance on the generated test set variations. During testing, all models receive seven in-context examples. Each example consists of an original GSM8K question and its answer, followed by mutated version of the question along with the corresponding CoT solution and answer. These examples cover different types of mutations, including two additional mutations beyond those discussed in Section 3.2 and Figure 1. To maintain consistency with the in-context example format, we also include the original question and its answer for the final mutated question that the model is expected to answer. Figure 20 illustrates a detailed example of the prompt, while Figure 12 presents the results. The findings indicate that most models perform significantly better on the generated test set variations when provided with both original and mutated examples, compared using only original GSM8K examples (Figure 2) or only mutated examples (Figure 19) as in-context examples.
To visualize the comparison, we summarize model performance across two dimensions: the average accuracy across all six test sets, as in the left bar plot (x-axis), and the standard deviation in accuracy the five mutated test sets w.r.t. the raw test set (y-axis). The bottom-right corner reflects the ideal balance of high performance on both raw and mutated sets. In this way, we consolidate model performance across these three different in-context example scenarios in Figure 8 and illustrate how to interpret the plot using the Llama3 model as an example.
Model Accuracies on GSMBK (8 Mutated In-context Examples)
055
95%
85%
Samplebalues
Uselessinfo
75%
Counterfactual
10%6
mentiandiiionnl al
85%
MddDependence
53%
52%
45%-
15%
30%
25%
15%
10%
Iamaz
Ilamaa
Iama3.1
Ulmaa3703
GPT-3.5
GPT4
GPT-40
GPh-1
ph3-mini
mhi..5mmii
phi3-cmall phiB-medium
phi4
Figure 11. GSM8K: Detailed accuracy for models in the Phi, Llama, and GPT familics. All models are prompted with mutated in-context examples. Check Figure 19 for the detailed prompt. Due to potential noise from the automatic mutation process, we performed a human evaluation on the mutated test set. We added the percentage of invalid examples in each mutation category to the top of each accuracy bar, assuming that these QA pairs were correct, the models would answer them correctly. This provides an upper bound on the model's performance.
Model Accuracies GSMBK (8 Paired In-context Examples)
2
93%
Samolevolues
85%
Uselesstnfa
Gountwhactoal
10%
menttCnnditooal
0374
AddDependence
80%
Row Error
52%
45%
40%6
35%
25%
25%
Aamw3.3
Llama3-705
G7T-3.5
GFT40
GPS01
ph3-mini
phit.5mmi
phitssmall
phis-medium
phi
Figure 12. GSM8K: Detailed accuracy for all models in the Phi, Llama, and GPT families. All models are prompted with 7 paired raw and the corresponding mutated in-context examples. Check Figure 20 for the detailed prompt. The missing Llama-2 experiments are due its failure to follow instructions and produce correctly formatted answers.
E.3. Composition of Mutations
The defined Ladder of reasoning hierarchy (Section 2) and the automatic variation generation pipeline (Section 3), open the possibility of combining multiple mutations to create challenging test set variations. In this section, we combine the Sample Values mutation from Level-2 with the remaining four mutations (excluding Bi- FauntterFactual) to create four test set variations, each containing two types of mutations. The models' accuracy can be find in Figure 13.
15

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
To visualize the comparison, similar Figure 8 Section 6.2, we summarize model performance along two dimensions: the average accuracy across all test and the standard deviation in accuracy on the mutated test relative to the raw test set. We combine model performance test set variations with single mutation and those with two mutations in Figure 14. As observed, all models exhibit lower average accuracy on test variations containing two mutations compared to those with a single mutation. The increased standard deviation of accuracy w.r.t. the original test set suggests that composing mutations expands the performance gap between the mutated sets and the original set.
Model Accuracies GSMBK (8 Raw In-context Examples)
Samotevalues
0774
uselessinto
Counterfactual
7055
635%
Nolloependeec
0374
4599
4055
25%
25%
Jamaz
Beema
Jema3.1
Unmab-703
GPT-355
GP14
GFT-4o
GPF-1
phD-min
phO.5mmm
phiD-smell
pho-mmedum
phil
Figure 13. GSM8K: Detailed accuracy for all models in the Phi, Llama, and GPT families. All models prompted with 8 original in-context examples. However, with the exception of SampleValues all other mutations are combined with Sample challenging test variations.
SampleValues Values to create even
Summary of performance on single mutaton and composed mutation
30.0
Single Mutations
X
Mutations+SSanie Sampleellus
Standard deviation of ACC w.r.t. Raw
27.5
GPT-4
25.0
X
lama3-70B
22.5
GPT-40
20.0
ma3.1
17.5
ama3
Xnhi phi
chid-smail
min-min
0hi3.5-mini
mediuum
12.5
GT
10.0
Average ACC (%)
Figure 14. GSM8K: Visualization comparing model performance on test variations with one two mutations, We summarize model performance across two dimensions: the average accuracy across all six test sets, as in the left bar plot (x-axis), and the standard deviation accuracy the five mutated performance on both raw and mutated sets. In this plot, dots represent models' accuracy test sets with single mutation, while crosses indicate their accuracy on sets containing two mutations. Model structures are differentiated by color, same structure represented by the same color.
w.r.t. the raw set (y-axis). The bottom-right corner reflects the idcal balance of high
16

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
E.4. Decomposing Model Performance by Number of Reasoning Steps
ACRRS Soa peienar)
NCCInO NarON THE piennee
A0W 37 for pluccedua
Sulaes 218644
Schlee VIAGA
Sanaag Wureer Tapysta CPP4
GAs MAg Watee Pan OFT
Figure 15. GSM8K: We analyze model performance across varying numbers of reasoning steps. We define the difficulty of numerical reasoning question by the number of calculation steps (operations) in its code solution. For each model, we compute the average accuracy across examples grouped by their number of calculation steps. In the plots, the x-axis indicates the number of reasoning steps, and the y-axis shows the accuracy. The results reveal that, across all models, even when controlling for the number of reasoning steps, performance is consistently lower on mutated examples than on the original test set. Furthermore. Level-3 mutations pose a significantly greater challenge than Level-2. Notably, models perform worse on Level-3 mutations with just three calculation steps than they do on original test examples requiring seven steps, indicating a substantial gap.
17

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
E.5. Detailed Prompts
[["role": "system", "content": "I will provide math question along with sample Python cade which solves the question. The names of variables in this code may be intentionally misleading. Then I will provide an updated version of the math question which differs from the original question. It nay include irrelevant information references to variables which are not used in the code. would like you update the code represent inclusion this additional sentence, so answers updated renove variables
question accurately. may you disregard the nanes the variables in the code and focus on the values they represent. Do not nodify which variable is returned at the end of the code. If you are unable to complete the task, please return the phrase
of code.
inportant
'Unable to complete'."" .")
("role": "user", "content": "Nath question: poriginal_text/lnnn
"Python code: Drriginal_code\ntl
"Updated math question: (nutated_text) text)
Runnable python with no additional text." "Do not include python beginning."")
"Requested output
Figure 16. Prompts used in GSM8K quality control. We prompt the LLM to back translate the mutated math problem into Python by modifying the original question's Python solution. The generated code must produce execution result that matches the ground truth answer of the mutated question.
18

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
[("role": "system", "content": "You helpful assistant."),
("role": "user", "content"; will provide a question along with the Python code that solves
will set particular variable in the code a different value. would like you to describe this change in natural language and offer brief explanation for it in single sentence. Math question: (origgial_questioon)nl Python code: (original_codef] _code)\n'n (variable_mmg [erraabeemmeee>>>> Describe this change in natural language.
(new_value).
OverWriteValue
The request format is: The change is that instead of... '")}]
[("role": "systen", "content"; "I will provide math question along with the Python code that solves it. will add irrelevant variable. would new variable which is related to the setting the word problem,
line of code which describes the definition
new, you create meaning for this
but ultimately irrelevant to the question. The change to the word problem should be single sentence in natural language, describing the new variable and its definition, which makes sense in context.
The change must not affect the rest of the word problem. Do make information is irrelevant,
any reference to fact that but ensure that is. not use the nane of variable from the python code; only describe the change natural language."),
("role"; "user", "content"; "Math question:
Original Python code: (original_code)) codd
add the 1ine (new_line) which defines the variable (new wariable).............. Anyn Describe this change in natural language.
request format is: \"Suppose ... .\")}]]
UselessInfo
[("role": "systen", "content":
"I will provide math question along with the Python code that solves it. will define new variable, and then add the value of that new variable to the definition of existing variable. would like you to describe alternate version of the word problem which matches the updated code. single sentence natural
The change
word problem should language, which makes sense in context. Make sure to explicitly state the value of the new variable and how it is used in the problen."), (isiiaal_questton)/kk.
("role": "user", :content"; "Nath question:
Original Python code: prerignnal_coee\Mne
define the variable (v2) be:
update definition of variable {v1} to: (vi_value)-
Describe this change in natural language.
AddDependence
The request fornat is: \"Suppose that ... in addition to A\")]]]
[["role": "systen", "content":
"I will provide a math question along with the Python code that solves it. I will add an if-else conditional to the code. I would like you to describe an alternate version of the word problem whose solution matches the updated code. The change to the word problem should be single sentence in natural language, without ambiguity, which makes context.
("role": "user", "content": "Nath question: belggiaalqqesttonn/al
Original Python code: (original_cmm careall__odd/nnnnn replace the 1ine {original_ __iiee] with the conditional statement coneettioon__eppaccceeeee Describe this change in natural language.
The request fornat is: Suppose that ... ... otherwise ... .")}]
InscrtConditional
Figure 17. Prompts used in GSM8K mutated symbolic representation to natural language.
19

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
"role": "user",
"content": "Question: Mimi picked up 2 dozen seashells on the beach. Kyle found twice as many shells as Mimi put them his pocket. Leigh grabbed one-third of the shells that Kyle found. How many seashells did Leigh have?"
{
"role": "assistant",
"content"; "Answer: Mimi has 2 12 <<2=12-24>>24 sea shells. An Kyle has x <<24*2-48>>48 sea shells. An Leigh has 48 3 c<48/3-16>>16 sea shells.
Aollss.nnus4 An#a## 16"
b
{
"role": "user",
"content": "Question: Frankie's s parents let him have many pets. He has six more snakes than he has cats. He has one less parrot than cats. six his pets have four legs. He has dogs. How many pets does he have in total?"
b
{
"role": "assistant",
"content"; "Answer: He has 6 <<6-2-4>>4 cats. An
He has 4 <<4-1-3>>3 parrots. An
has <<4+6-10>>18 snakes. \n
He has total of 2 + + 18 c<2-443+10-19>>19 pets, An#Hi 19"
more examples
"role": "user",
"content": "Question: mid-May the depth of river in Moreland measured By mid-June the river is 17 feet deeper than mid-May By mid-July the river is times deeper mid-June If the river the river in mid-May ? Let's think step step. At the end, MUST write the answer after "#ume". that, this might trick question designed confuse with additional information. Look for irrelevant information, new assumptions or distractors the question.
feet deep mid-July how many feet deep was
Figure 18. Prompts used in GSM8K lesting. The prompt contains 8 in-context examples.
20

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
"xole": "user",
reade pages day Belinds reade pages day, aesume Belinda accually reads Peges a day, how neny more pages does danet: need in deys?"
"content"; "Question:
Frole"; "aseistant",
"conteat"; "Answer: First Hind the number of pages Belinca actually reads per days b0 pages/day, then tind the total di -ference in the number of pages Janet and Bel belinda read eash days 80 pages/day 50 pages/day = cce0-50-5003d pages/day Then multiply the deily difference by the namber of days to Eind the totel di difference in the number of pages they reads 30 pages/day
daye 6030-61100318 pages 180"
rzll"" "uase",
reads pages day indi pages day, Suppose Rellinda only if Janct reads nore than 100 pagzo a days it she roado 130 pages a cay, Bclindo does not read at all. Bov many more pages cocs Janct waa in 5 daya?"
"role": "asointanc",
"conteat"; "Answez: First find the number of sagea Belinda actually zeada pez days since canet reads EO pages per day which is leas then 100 pages, Belinds reads pagea/day. Then find the Lotal differemce in the numbee of pages damet and bel Belinds read days pages/day pages/day sc83-0-10580 pages/cay Then multiply deilyy disterence by the or days to find the tocal diffarance in the numbar pages they read: ED pages/day days <x80* pages stef 480" b,
"role": "user",
"contant": "Questiins law now ane lawn and therges $33. Last week be awwedd 16 Lavns and Lhrwe uuxtorerx math gave 610 tip. Assume Lee meeds to pay the threa cuatonara $10 each since he danaqed their lavna inatead of qetzing $10 tips. dow nany dollars did Lee carn moxing lavns last Weak?"
*rala"; Passiitantt,
"conteae"; "Anzwer: 33 A 16 32209166-3889222 3 10 9003-10-003300 520 30 5<<328-30 498>>4999 Lee carned $498 last
noxing
438"
"zole": "user",
nowed 16 Lavne three custoners tip. Ascume Lee's neightoz aiso noved 16 lawes last week, How mony dollars did carn moxing Lamne Last week?"
"conteas"; "Question: lee
laxn and charges 633.
"role": "asuiatanc", "content"; "Answers
56c33'1 6-5283528
9063-19-503330
$c0529-30=5583355 earned
nowing lowns last voek. tit: 538*
"sole"; "uscr",
"content"; "Question: lee novs one lamn and charges 933, Last week he nowed 16 Lavne and three customere each gave him 610 tip. Assune Leo get payed in pounda not dollars. How many pounds did Loo carn nowing lavna last wook?"
"rolo": "assistanc",
*contant"; "Answeez 33 + 16 84588088 3 10 Scc3-10-300033 528, 30 Scc528: 30=55250558 Law earned $558 noxing Lonne last veek, t#tt 538*
"sole": Puser",
"content"; "Guestions tee one lawn and cha $33. Tass week he nowed 16 laune and three custoners each gave him tip. Ascume Lee get extra banus reom his company 95, fox each customes served. many dollars Lee carn moving lawns last woek?"
"role": "assistanc", "cor tent "Ansvers 33 accasel 65578 3 10 $cc3-10-5503300 Bocc* Soat 598 30
Scx528 (301 15-5739573 Low warned 9575 nowing leara last woek. 4644 573"
"role": "usee",
"content"; "Question: lee nove one lawn and charges $33. Tast week he nowed 16 lavns and three customers each gave him
910 tip. Aszune Lee did not get tips fron his custoners. Hox nany dollars did Les cazn nevina laxts last wook?"
b,
"zole": "assintant",
"conLaat"; "Answwez 33 + 16 8cc35*16-5200522 lawe wazone 6528 musing lavma laxt awuk. +444 528"
"role": "user",
"conteat"; "Question: Bill had to finish a project fron wozk that was to take hin 4 days. Be took 6 aeven-hour naps in being calculated spord working on the project? Let's think step by step. erd, you SUET
the tine spent naps is aws haps and suzation each nap, long write the answer after '898s", Note that, this night be a tztck caeation designed to confuse LLM with additional information. Look for irselevant information, nex accimptions or distractors in the question.
days,
producs
Figure 19, Prompts used in GSM8K testing. The prompt contains 7 in-context examples demonstrating different types of mutations.
21

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
12 preanble or systen instsuctdon
As an expest peobion sakvee, ankie acep by stup mesated nath quartion baaed as the oeiginel queationn and its azamez. Rote that, this night beick gamaiio dexigned contuze luo witl cotilimaal intosmacion.
uasuptien dhatsactoss queatain. answee
you RUSt
12 shot-s
The original queatiens ze Jaret reado pages day and selinds reade 30 pageo a day, how many nore pages does Janett read in 6 days? giestion shove 151 330 Sheett rend queertions Janet reads pages day and
The answer
rerds 50 pagne day, asssre Rell scroslly reade pages dey, aany none pages
The staves Lie quastion above las Fiest find the nunber of pages Belinds actually ceats par days 50 papearday. Then Eind taw total discezence in the numbur DE
pages danes and Bellens read each days 00 pagesiday - 10 pagee/ta ccan-10- 500000 pageoiday Then nultiply the dailyy ot etfeeennce by the nomber of daye to rind the Lotal aitiurmnce on Lae cumbes ef pagea they ceats 30 papeardayy Y daya = 6030-602500010 pagus aide 18c
ahot-2
queations Jaret reudo pagea day selinds seads 30 pages day, how miny pages does danet read in cays? 300
answer the giestion
the watated queection bor Ef Janet reads 60 pages day and ne belinda reads 30 paged a day, supposs that Solinds only reads ST Janet reads wore than 160 pages a dogs
1f she reade 100 pages a day, nel inda does not read all. sow vany wore pages doed Janet read In dagss
The answer the giestion shous tss Piinet sind the nomeer of pages Re teda scenallly by meade per days stree Janen reads pages per day sho ch to tees than 1on pages/day. miltiply datly difference by co85h--tttbocie pages 405
pages,
total
pager Janet defference
reed each days pager/seg pages/tay of pages they
dage
pages/day days
f: shot-5
the original queations Lee acwo one Jesn and charges 633, last week be eoved 16 Loxta and cheee customers each gave his 910 cip, bow vary dollaro did Lee eare
nouine isons lass weaks
answer queeet
guestion showe 558
the
and charges 933, last woek hee nowed 34 laanss and thrre mistowers each gaze his 410 Ansime Lee needs to pay the three custosere 515 each since he dasaged their laone fnsteed of
210 ripe, Fox taae did the earn incting 1ains 1ast week?
the anewer of the guestton shous 1el 53 + 16 - Soe53 5-5800522 3 x 12 Sectel Secteld-stasn d-stasto 550 - 53 9cc788-3--4445ot8 has earned pes8 rosiing isems last week.
test <sn
scvn
chazges 533, lasz
theee custosers each gava
rany dollaze
noving lovne last weak?
the answer of the giestion above ist 558
the astated cuartion: Lns zova one leve and charqus 833, last waek ha nowed 28 Lasna and these cistoraca each gare his 510 táp. Axaue Leets ta melatbor alae cowed 16 lames last wnek, Ho wany do'l arre and tee earn "eeuting haans lest vence
the snawes the quastion above las 16 5rd352242880225 22 D0001224520000 528 52 300522133-0299938 ies warmed 9238 maving Lecnu Last verk.
shot-5
The original queation: See news one 1avn and charges 533, last weak he roved 16 Laima and thrse custosers each gave his $10 ctp, Row rany dollare did Cee aarn
coving lsvnz last weak?
the anawer the quastion abovs iss 530
custoeaca qare tip. Asause est
ausy
sovine
12 ao02r:00220000 528 52 300528/3000800508 ias uxrmed 5238 maving Levna Lest yeak. the stawer the quastion above das 16 215332285299822 dess 528
zbot-6
The uriginal gieat uaw aceo une Loom and charges 633, Lun. wock he noved Luvau and cieee curtumuss wach gare his 910 Lip. Boe nany dollars did lew wasn
the answer the quastion above iss 550
lavn and charqes 633, last week ha noved 2t Lakna and three custonaca each qave hix 510 tip. Anaume Lee qut extia banua fron his coepany 25 for each cratceat he served. Boe rany dollara did See sazn novine Levna lass veak?
the astated cuartion: lns rove
The anaver of the ques tice above isz 33 + 16 5ec33 12522802225 3 22 w 20020220 3 x pora1e41500s 528 a 30 15 5005221301:555100055 Las eacned 1512 nowing Lasta last meck. dedd 573
aboi-1
The uriginel quealiens Sau aour 1om and charges 633, luz. eook te soved Luvau and Jhew custurers vuch gave his 915 Lips bor eany dollazs did See noving Lovns Las. seak?
The atawer of the questios above las 598
the astated cuarsion: las ross one Laom and charqes 633, last week ha nowed Lasna and thrse custonasa each gare his 510 tip. Anaume Lee did not get tips custowers. aany dollars did lee earn nowing question
Heek?
5ex33 3152289522 sazned noving lasna lest veak,
17 carqut queation
The uriginal queatiens in u cuke. Maztin ansveced thewe cevur quvations cosrectly than Salauy, and Salawy ansveced wigh. noce queationa cossectly than cumpeel.. Conpbell anncened 35 queettone correctly. Row may did varein answer cornectly?
the stawer the Lis abore las guia,
answered ataessed guesticus correctly Cargoel. instead night queations. many mstated qusetion: questiora correctly. natils answer correct.yy the answer of the giestion sbove 151
qusetione correctly than Telsey, Kelesy unavered aight more quasssone cornectly Carpbell.
Figure 20. Prompts used in GSM8K testing. The prompt contains 7 in-context examples. Each example contains raw question and answer, with the corresponding mutated question and chain-of-thought fo the mutated question, with the final answer. The 7 examples are demonstrating different types of mutations.
22

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
F. Appendix: CLadder Details
F.I Benchmark Details
The CLadder dataset (Jin et al., 2023b) provides a systematic evaluation of causal reasoning abilities in LLMs. It consists of 10,000 causal graphs of binary variables (i.e., Bernoulli conditional distributions) that encompass common treatment-effect estimation scenarios, such as confounding, mediation, and collisions. Each causal graph is associated with multiple queries, spanning the three levels of Pearl's 's ladder of causation: associational, interventional, and counterfactual. Each dataset example includes (1) a causal graph, (2) a query, (3) a causal engine that computes over the casual graph and the query to get binary answer, and (4) a template-basee based formulation that translates the causal graph and the query into a natural language question for the LLM interface with. The benchmark evaluates models' causal reasoning abilities by requiring them to (1) extract causal concepts from natural language and (2) apply causal inference (either implicitly or explicitly) over the causal concepts, such as do-calculus, to calculate the final answer.
F.2. Pipeline Details
F.2.1. QUESTION TO SYMBOLIC REPRESENTATION I
We begin with filtering pass of the 10,000 examples in CLadder, removing all examples corresponding to query types "backdoor adjustment' and 'collider bias' (in total 1,747 examples). The former was filtered due to the inherent complexity of the python it would require, and the latter because it is only present for one type of causal graph structure. This leaves 8,365 examples from CLadder after filtering.
Filtering:
Next, we develop a parser to automatically translate these examples into Python code with the help of the causal engine provided by CLadder. The parser determines the query type from the meta data in the CLadder examples, and extracts the relevant variables from the natural language questions. For every query type, causal graph, and estimand, the parser generates snippet of executable Python code which computes the estimand.
Parser:
To check that the NL to Python code parser is successful, after conversion of the examples, we execute the Python code and compare the computed estimand value with its ground-truth in CLadder. 60 examples showed a discrepancy, leaving us with 8,305 examples (99.34% coverage of parsed examples).
Validation:
F.2.2. SYMBOLIC MUTATIONS 2 AND MUTATED SYMBOLIC REP TO NL
To examine the robustness of LLMs in handling variations in question phrasing and irrelevant information, we include two key mutations:
UselessInfo: We add extraneous information at two levels. (i) Relatedlrrelevanttffo With Meta-Llama 70B Instruct,
generate sentences for each question that semantically related but causally irrelevant. (ii) Unrelate- dIrrelevantnffo: We prepend the natural language description from randomly selected CLadder example to the beginning of each question, ensuring that the added sentences remain semantically irrelevant to prevent ambiguity. Both RelatedIrrelevanlng and Unrelatedrrrelevantien describe Level-2 mutations.
CounterFactual: We modify the probability values the original question by introducing a counterfactual assumption, such as: "Suppose the probability of <Event> <new <new-values the counterfactual question, we update the corresponding probability of the conditional distribution Python code and execute it. CounterFactual is Level-3 mutation.
value> instead.". To generate the ground-truth answer for
In Table 5, we provide an example for each type of mutation. Additionally, we generate two more test set variations by combining the FauuterFactual mutation with each Uselessinfo mutation. The full results are shown in Figure 21.
23

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Mutation
Problem and Buyesian Network
Probability distributions
Code
Original
For hushands that don't set the alams wives that don't the alams the probability ringing alana
H (busbard sets alams) (wite alam) (alamm rings)
Probabil lie
0.08
punct gigeen, hotot
p. giv.gn not
8%. For hashands that don't set
Punugi 0.54
the alarm and wives that set the
alarm probubility of ringing
P(A 1 U. W 1) .54
P(A 1 Q. 0)
pin.ggiven_h hunet_ 0.41
a.net_a_given_nee
- p_e_given_rot_ha
husbands the alarm and wives that don't set the alarm, the probubility ring- 41%, For hushands
P(A H 1. = -0) .41
P(A 1 0.
pua_given.
0,86
P(W 1|8
Punot pett_aa.g w.g.eeee bure h.x =1 P.A.given
P(W- 1|8 = = =224
0.74
alanm and
punet givven not.h
p.M.givee_not..
the alarm, the probubility of ring- ing slams 86%. For hashands that don't
Estimand:
P. P.M_given_h 0.24
alarm, the prokabilliyy of alarm set by wife is 74% For hus- bands that set the alemm, the protu- by wile 24%, disregond mediation through wife, would lusbond posi- rively affect alame clock?
P_not_x_ - P_M_givvn_h
M[xx=1 V2=00
For
bility of alarm
VIE
Σ >
tem_ F_noo v_g.ven__ott_. _ (p.a_given___not.e
p_a_given not _ot_ nonot_v
JM(Y - 1
For 8--
P(Y = 1|X = V2= v)
tem_1 - B. given not h giv.nive__.vv
Pinel sun recult tezm_0 + comn] it result > 0: print ("yes")
clec: (ront
CountefFacuual
For hushands that don't set the
P(A 1 0. W 0) 255
Probabilities
alamm and wives thit don't set the
P(A = 1 W 1) .54
p.a pa.given.noo not h.not 0.25
alurm the probubility alarm
P(A 1
.41
set by wife 24%, Assume that
P(A H O.W
puL_otven_pet_uu 0.54
the probability the alara ringing if both the husburd and the wife
P(W 11H1 00) .74
__st____ivra_nott
s_a_oiven_rat_hh
P(W 1|H = ==== .24
LuLesvven_hoooo 0.41
alarm, 25%, disrogurd the mediation effect through wife. would busband positively affect alana clack?
p_oet_a_ = Paa
pon-givnn 0.36
P_A_a
P_M_aivon_npt__) 0.74
= 0.24
P_nst_x_ s_v_oi
Por x
tem_0 p_not givee.entete (gLaggiven
Por 6-1
being] s sp
Pinal sun
result serr]]
swault De
print ('yes"t
elaes
("no*)
Inclevartlafo
hushands that don't alom and wives that don't i sei the alomn [..] the probubility of alum set by wife is 24% Assume that going to school each day is 80%. we disregard the mediation effect through wife. would hushand positivaly affect alamn clock?
(kids school)
(identiool) ical to original)
P(A H 0, 0)
P(A
0. G.W
probobiiiyy
P(A = H = 1.W =0) .41
P(A H O.W =0)
P(W 1 0) .74
P(W 118 1)=
Table Mutations in RE-IMAGINE for the Cl Cladder dataset. In the Bayesian network, gray nodes indicate variahle whose probahility
mass function (conditional on variables from incoming edges) has been intervened on or added as part of the mutation. The probability distributions for the irrelevant graph Irrelevantnffo are not stated. Note that the probability of the positive event specifies the full
conditional distribution since all distributions are Berouuli (c.g. P(A = 1|H = 0, W = 0) = 0.08 - P(A = 0 = 0, W = 0) 0.92).
24

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Model Accuracies on CLadder
Raw
UnFelatedirrelevnntino
Relatedireleeanttnnoo
CounterFactual(CF)
CF+ E-UUnelatediirelevantnno
CF E--Reatedirreleeantnno
Composing two mutations
Accuracy (%)
65%
60%
55%
50%
Llma--388
Uama-3.3-708
GPT-4o
GPT-o1
Phi-3-ming
Phi-3.5-mini
Figure 21, Cladder: Detailed accuracy for models in the Phi, Llama, and GPT families.
E3. Prompts for mutated questions
Below are illustration of NL prompts for our mutations on CLadder. We highlight parts of the original question red, and the mutated natural language in blue. The prompt consists of the System prompt (stated first below), followed by mutation-specific prompt (stated thereafter).
System Prompt We the following system prompt throughout the evaluation.
You are expert causal inference and reasoning. You will given quest and must answer with "yes" oz "no" only.
Related UselessInfo
Imagine selt-contained, hypothetical world with only the following conditions, and without any unmentiones factors or cousal relationchips: The probability of the hushand being axaka when the wife gats the alarm is 858.
alarm, probability direct effect on wife and slarm clock. Mife has a direct effect on alarm clock. don't
huaband
wife being
good
733. Husband
For husbands
probsbility ringing alarm is 114. For husbanda that don't set the alarm and sives that set the alarn, tha probability of ringing alarm is
alamn
don't set
slarm,
60%, For huabanda that the alar wivea that don't aet the alarm, the probability of ringing alarm fa 468. For husbands that set the alarm and sives that set the alarm, the probability ef zinging alarm is 923, For husbands that don't set the alam, the probabil'ty of alarm set by wife is 618. For husbands that aet wife Does husband positively affect slarm clock through wife?
alarm, the probability
Unrelated UselessInfo
Imagine contained, hypothetical world with only following condiLions, without any unment ioned factors or causal selationshipe: The man in the zoom has direct effect on reem. The candle has direct effect on Them.The overall probability the cardla and dark room is 124. The probabil'ty of blowing out the candie and dark room ia 518, Tmagine self contained, hypotheticas world with only the foilowing conditions, and without any unmentioned factors causal relationships: Blusband has direct effect wife and alarm clock. Wife has direct effecs alazn clock.
blewing out the candle
probability not blowing out
set elarm, at. For husbands that don't set the alam and wives that set the alarm, the probability of ringing alarm is
husbands
probability ringing
is
wives that don't set alarm, the probability ringing alarn ia 418. For husbands that set the alarm and wives that set the alarm, the probability of ringing alarm is 868. For husbands that don't set the alazn, the probabillty of elarm sez by wife is 743. Por husbands that set the alarm, the prebability of slarm
54%. husbanda
the
wife is 248.
If we disregard the mediation effect through wife, would husband positively affect alarm clock?*
CounterFactual
25

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Imagine a self-contained, hypethetical world with only the following conditions, and without any unmentione factors causal ralationsh!ps: Rusband has a direct ef wi and alarn clook. Ki has rert effect on alaza clock.
For husbands that don't set the alamn and wives that don't set the alarm, the probability of ringing alarm is 118. For husbands that don't aet the alazm and wivea that set the alarm, the probability of ringing alarm is
603, Poz husbands that set the alazn and wives that don't t set the alazm, the probability of ringing alazn is the alarm and wives that set the alarm, the probability of ringing alarm is 928. For husbands that don't aet the alam, the prohability of alarm set by wife is 613. for husbands that set the alarm, the probability of alarm set by wife 14,
468. for husbands that
Suppose Probability of singing slarm, given that alarm did not set by husband alarm set by wife is 0.46. Uses husband positively affect alarm clock through wife?
Unrelated UselessInfo x CounterFactual
Tmagine a sel f-contained, hepothet world with only the foll lowing cond't tiona, and without any unment loned factors or causal tonshipsh The man the room has direct effect coom, The candle has direct effect on roem.The overall probability of blowing out the candle is 68t. The probability of not blowing out the candle and dark room 124. The probability of blowing eut the candle and dark reom is 516. Imagine secontatainedd, hypotheticsl world with only the fol loxing condi and without or causal relstionships: Husband has a direct effect on wife and alarm clock, Wife has a direct effect on alam clock.
unmentioned factors
For husbands that don't aet the alamn and wives that don't t aet the alarm, the probability of ringing alarn la 118. For husbands that don't set the alain and wives that set the alara, the probability of ringing alarm is
604. For husbands that set the alamn and wives that don't set the alarm, the probsbility of ringing alarn is 466. For husbands that set the alarm and wives that set the olarm, the probability of ringing alarm is 928. of alarm set by wife is 618. For husbands shat aet the alarm, the prebability of alazm set by wife is 14.
For husbands that don't set the alarn the probahi
Suppose Prebability of ringing alarm, given that alarm did not set by husband alarm set wife is 0.46. Does huaband poaltively affect alarm clock through wile?
Related UselessInfo x CounterFactual
Imagine aell cont-nonnained, hypothetical world with only the following conditions, and without any unment ioned factors or causal reletionships: The probabillty of the husband being awake when the wife sets the alarm is good mood is 734. Husband has
a56. if the husband sets the slarm, the probability of the wife being a direct affect on wife and alarm clack. fife has a direct affect on alarm clack. the alarn and wives that don't set
For husbands that don't
alarm, the probability of ringing alarn la 118. For husbands that don't set the alarm and wives that set the alarm, the probahility of ringing alarm is
60%. For husbanda that set the alam and wivee that don't t aet the alarm, the probabil ity of ringing al alarm ta 468. For husbands that set the alarm and sives that set the alarm, the probability of zinging alarm is 928, For husbands that don't set the alazn, the probability of slarm sez by wife is E14. For husbands that set the alarm, the probability of alarm set by wife is 14.
Suppose Probabf lity of ringiing alarm, givan that alarm did not set by hushand alarm set by wife is B.46, Does husband positively affect alarm clock through wile?
F.4 Prompts for generating IrrelevantInfo mutations
To generate the NL question for the IrrelevantInfo mutations, we use the following auxiliary prompt. The template <question> refers to the part highlighted in red in the prompts stated in Appendix F.3.
System Prompt to Generate Irrelevant Information
You are tasked with generating irrelevant probsbi atatements to enhance causal reasoning questiona, These statements will be added to the beyinning of the question, right after the pazase: "Imagine self-contained, hypotheticsl world with only the following conditions, and without any unmeationed Sactors causal rolationships:" * The irralevant probahility statements must adhere the following guidalines: They ahould describe probabl described in the question. The prosabililies should be resllatic the causel reasoning task or relationshigs in the question. They mast blend seamlessly into the hypothetical context without introducing new causal relationahise. distract from solving the cere problem. Structure of Irrelevant Frobability Statements: Use probabilisies or percentages (e.g.r "the probability of x is yt*). include conditicnal probabilities where appropriste le.g., "It occura, the probability of ia (t*). Ensure the statementa slige with the general sone of hypothetical. worlds while remaining inconsequential to the reasoning procesa. Your Task: Generate two Lerelevent prcbability statement for a given question. Ensure they are consistent with plausible, end add complexity without affecting the causal reasoning. Focus on adhering to the structure and requitements outlined above. You must ONLY give the two stalements as output and nothing else. DO NOT star. with phrases like "Here two irrelevant probability statements for the given question:" " <question>
or conaltional probabt littes related to entities, actiora, or settings
plausible but should have irpact on
They should add complexity to the quest ion but not
context,
26

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
G. Appendix: CRUXEval Details
G.1. Mutations
We summarize the mutation types implemented for CRUXEval in Table 6. Note that each of the five code mutations be implemented as both Type-2 and Type-3 mutation, depending on how the resulting question is posed; the corresponding two prompt templates are presented in Appendix G.2.
Mutation type
Description
Original Code
Mutated
Selects random opera- tor of type Rnmp. ant.UnezyOp, ast.BoolOp,
Replace Operator
def (Lext, lower, uppez):
def [Leext, lower, uppec):
count
count
or ast Aut.AAgnesig for replace- Replaces basic arithmetic operators with their inverses (e-g- addition with subtraction). Flips boolean operatoes and unary operators (e.g, "and" with "or"). Replaces comparison operators their negutions leg. "in" with "not and < ">
list
new_text
for char in text:
for char in text:
char
char.isdecimal
char.indecimal
else upper
else upper
char ['p', [C]]:
if char in !'p", (61]:
count 1- new aww_texx.-pppedddcee count, (oi
count 1- new_text www_textaappandcchaa (chart count, 1 ".joontnew_text
Selects random string instance for replacement. Replaces the string with random sequence length.
Mutate String
def [ttext, lowez, uppez):
def fitext, lower, uppes): count new_lext
count =3
new_text list()
for char in text:
for char in text:
char else upper
char.iadacimal
char.isdacimal
else upper
if char [Tp', 'C']:
if chat in ['E', 'C'::
nex_text deext.eppppdde appand zeturn count, " jofo (new_tten))))
nex_text. append (char)
return count, 1 "."ofn(new_texx
Selects a random instance of type bool, int, float for replace- ment. Boolean values are replaced with their negations; integers perturbed by uniformly random integer hetween 10 10; floats are perturbed by uniformly random nonzero floa between 10.
Mutate Value
def [aums): out - []
def [(nums)
output -
for aums:
for n nums:
output output.appennt append (nums. count in), n))
putpat.appente ((nums. count (2), n))
output (revarse Truo)
outpus scrt (reverse-Fall
return output
return output
Sclects a random conditional node replacement.
Swap Conditional
(t!l: for in t:
((t): for in t: if not c.ismueeicc(): return True
If both an 15 and else branch are present, code body for each branch swapped.
if not c.ismumeric():
return False
If only an 1 branch is present, the condition
else:
else:
branch negated (if x becomes LE not x).
False
Selects a random function call replacement (attribute function calls are not included). Defines a new wrapper function which calls original function, and replaces the original function with call the new function.
Redefine Function
def (dic):
def xxxx (arg0)
for k, sorted (dic (die.items items
return List((rg0)
4
key-lambda xt
lea fstz 1]:
def [dic)
dic.pop (k)
for k, in sorted (dic. key-lambda x:
(dic. itlmm
len ( [=-1]
pop (k)
return (dic. stimc.temm// ()
Table 6. Mutations in RE-IMAGINE for the CruxEval dataset.
G.2. Prompts
Below list the prompts used in testing CRUXEval. All models are tested using zero-shot prompting, only.
Prompt for Level-2 mutations:
27

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Consider the follewing code with missing value represented by 1221: (f.question)
Based the given Python code, which contain errors, complete the assert statement with the output when executing the code ou the given test case. Print only the exact text to zeplace "2?" the assect statement, make the assert statement incorrect or incomplete.
Lo NUT cutput any information, if the function
Prompt for Level-3 mutations:
Consider the follewing code with missing value represented by 122':
(f.question)
Suppose change is now made to the code, as described by the following diff: (diff)
Based on the given Python rede, which may contain errors, complete the assert statement with the putput executing the ande on given test case. Print only the exsct text to replace *??" the sssert statement, the assert statement true. Bo cutpot any extra information, even if function incorrect Incorplete.
G.3. Coverage
Not all transformations are applicable to all problems; we report coverage statisties below. Mutations are performed on 800 total factual examples, with 88.9% of factual examples covered by at least one mutation.
Mutate
Mutate
Redefine Function
Replace Operator
Swap
String
Value
Conditional
30.3%
56.6%
54.1% %
50.6%
42.5%
Figure 22. Mutation Coverage Statistics (as percentage of total CRUXEval data)
G.4. Evaluation and Matched Factual Accuracy
We plot the accuracy on the factual and mutated CRUXEval benchmark for ten language models in Figure 21.
Model Accuracies on CruxEval
Raw:
50%
Mutate String (L2)
15%
Mutate Value (L2)
40%
Recieline Function
Replace Operator (L2)
jocuracy
35%
Swap Conditional (L2)
String (LJ)
25%
Mutate Value (L3)
20%
15%
Rep Operator (L3)
10%
Swap Conditional
5%
0%
Liama 3.1 (8B)
GPT-3.5
GPT-4
GPT-1o
GPT-o1
phi3-mini
5-i3.5-mini
phi3-small phi3-medium
phi-4
Figure 23, CRUXEval: Detailed accuracy for models in the Phi, Llama, and GPT families.
Because the coverage statistics for each mutation fall well below 100%, each mutation is tested on only a subset of the total CRUXEval benchmark problems. We therefore consider the possibility that there may be correlation between which mutations apply to given problem and the difficulty of that problem for the models evaluate. In order to account for this fact, we also compute matched factual accuracy scores for each mutation. These scores report the factual accuracy rate of the model when tested on only the same subset of problems to which the mutation applies. For example, in Figure 24, the green bars represent accuracy scores on the factual, Level-2, and Level-3 variants the 56.6% of CRUXEval problems which admit a Mutate Value mutation. Note for ease of reading that the palest bar of each color in Figure 24 represents the matched factual score for the corresponding mutation.
28

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Model Matched Accuracies on CruxEval
50%
45%
Accuracy (96)
40%
35%
30%
25%
20%
15%
10%
5%
GPT-4
GPT-4o
GPT-o1
Llama 3.1 (BB)
phin.5-mini
phi3-small
phi3-medium
Muratt Value
Redefine Function
Heplace Operator matched factuel
String Nutete String (L3)
Swap
Murate Value metched fectual
Replace Operator (L2)
Swap Conditional (L3)
Nutste String macthed hacruel
Redefine function (1.)
2000 Replace Operator (L.)
Swap Condition matched factual
Figure 24, CRUXEval: Detailed accuracy and matched factual accuracy for models in the Phi, Llama, and GPT families.
We observe that the overall trend of decreasing accuracy with Level-2 and Level-3 mutations is still evident with respect to matched accuracy scores. Nonetheless, we note that matched factual accuracy is consistently lower than raw accuracy on certain mutations; most drastically for Mutate String. This indicates that problems which contain mutable strings tend to be more difficult for LLMs than other CRUXEval problems, underscoring the importance of viewing matched scores for the fullest picture model performance. The further decrease between matched factual scores and the corresponding Level-2 and Level-3 mutation scores indicates that the difficulty-Iee correlation cannot account for the remaining loss in accuracy post-mutation.
H. Appendix: Loop Details
In software verification, automatic inference of loop invariants is classic problem (Si et al., 2018). Since this problem is undecidable in general, many heuristics have been proposed, including those based machine learning. Recently, LLMs have been demonstrated to perform well on loop invariant inference of integer programs (Wu et al., 2024; Kamath et al., 2024). In this paper, we mutate such tasks and evaluate the efficacy of LLMS on the mutated tasks.
Each task has program with loop and an assertion. The goal is to infer predicate that satisfies the following three conditions: it holds before the loop starts executing, holds for each iteration of the loop, and implies the assertion when the loop exits. Finding such predicates can be tricky even for small programs. Consider the task in Figure 6; the predicate 2 satisfies the first and the third condition but fails to satisfy the second. As another example, consider the simpler program
x=0; while (x< 100) x++; assert x==100
Given this program, the goal is for an LLM to produce the loop invariant & S 100. It is easy to see that & < 100 satisfies all the three conditions specified above. Note that there are infinitely many predicates that are variations of S 100, e.g., S 99 z 100, that are valid loop invariants and the model succeeds it infers any of them. Although inferring such loop invariants is undecidable, in practice, checking whether given candidate invariant satisfies the three conditions be done well by automated software verification tools like Frama-C (Correnson et al.).
To evaluate a model on an original mutated task, the LLM output checked by Frama-C that internally uses SMT solvers such as Z3 Moura & Bjömer, 2008), alt-ergo(Aii (Ait El Hara al., 2022) and CVC4(Barrett al., 2011). The model succeeds on the task if Frama-C succeeds to verify the LIM output as a loop invariant, and fails otherwise. Hence, a model can fail on a task for two reasons: either Frama-C declares the candidate as invalid, or the candidate invariant is valid but Frama-C could not prove its validity within the provided time bound. We evaluate on 250 tasks from Kamath et al. (2024) which have the property that Frama-C C succeeds verifying that the invariant for the original program also the invariant for all the mutated programs.
29

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Unlike GSM8K, applying Level-3 mutations these tasks is difficult. Once we change the values of the program variables, loop invariants cease exist. For example, if we mutate the example ahove then we can get the mutated program
x-101; while (x< 100) x++; assert x==100
with new initialization. There is loop invariant which discharges the assertion as the assertion will get violated when the program is run. Hence, limit ourselves category of Level-2 mutations that add useless information to the tasks in the form of additional variables and operations that leave the values of the variables in the original program unaffected.
Table 7 shows how various mutations operate the program in Figure 6. Figure 25 shows the results of all the models consider on 250 loop invariant inference tasks from Kamath et al. (2024). We use the following prompt from (Kamath et al., 2024), reproduced here for completeness.
You are helpful AI software assiscant that reasoas about how code behaves. Given program, you can Lind loop invarients, which then used verify property in the program. #rama-C is a software verificatien tool for C programs. The imput to Frama-C in program file with ACSE (ANST/T50 Spect ficstion language) annotations.
for the given program, find the necessary loop invarisats of the while loap to help Frama-C verify the post-cond:rion.
Instructions
Make
of the pre-conditions variable assignments in the progrem.
Analyze the loop body and make a note of the loop condition.
Output 1oop invariants that
(3) before the Icop execution, loop
111) avery iteration
fill) after the loop termination,
such that the Loop invariants imply the post condition. conjunctien, split it into parts. Output all the loop invarianta in one coda block. For exarple: ...
If loop invariant
/+3
1009 invariant il;
100p invarient 12;
./
...
Rules:
not variables Lunctions
not declared in prografl..*
++Do not make any assumptions about functions whose definitions are not given...
ssAll undeCined variables contain garbage values. Do not use variables that have garbage values,xx
AxDo not
keywerds that not supported in ACSL annotations for locps.
+everisbles that are not explisitly initialized, could have garbage values. Do net make any assumptions about. such values.**
ArDo not use the (X, Frel notation
any variable x..* x+Uo not use ron-deterministic function calls... Consider the follewing c program:
code ...
You are alwed to use implication to Lake caze of the conditional nature of the code. Use implication (#1 instead using ih-then.
For all variables, add conjunctions that bound the maximum and minimum values that they can take, is such bounds exist.
variable always equal smaller or larger another variable, joncitction their relation.
If the assertion is guarded by condition, use the guard condition in implication.
cortain variables ion-determineetic the baginning end of the loop, implication make the Invariant trivially true at that lonation.
Output the loop invariants for the loep in the program above. Let's think step by step.
30

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Mutation
Description
Code
Intervention Categories
Vanilla
Base Snippet with loop and assertion.
None
int. 1,
while 110000) (x+ y++2)
7/2 asseet y)?
Junk Hint
Adding junk variahles that disjoint from original variables and named _ and J.
Add inrelevant Info
int y Or int jank_c - 2, junk_1 whill 1000000 ixt-ys y4-z
Junk_] Junk_o 687;
1/V8 assest y2s
lunk
Adding 2 new variables that are disjoint original and named unremarkably (g0 and gl). Create new state- using randomly pling of new variables and con- stamts.
Add irrelevant Info and renarne nodes
int 1, D1
1, 1y 160000) : x y
90 178, 22 e0 5871 3/VE aesert
Read Original
Read the original variablee the code into the newly intro- duced oncs. Randomly ple original variables and oper- add ables. The arithmetic expres- sions updates to new vari- ables the original pro- gram variables along with the variables randomn stants. This mulation intro- inal program variables but writes to them.
Add irrelevant Info and rename nodes, dummy relationship be- nodes
while ly 100000] : x- y-+z
90 178 1/78 assert
q: 4. 637
orig-
Write Original
Increment the original vari- ables by algebruic identities of new variahles that equate 0. Randomly sample new vari- ables ideatity. Split con- stants increase complexity.
Add irrelevant Info and nodes, dummy relationship be- tween nodes
1,
int g0 1, 21
while ly < 1000001
(1120 gl1 -1+g0+g1 -2+g0-g11
16g0-g0 gll-g1
1eg0>g12;
x -yt 1 ligl-gt g0xg01 -1gl (1 + g
91 + g0) -1-g1-g0 -2xg1-g01;
1987
68%;
X
Increment the original vari-
Add irrelevant Info and rename dummy relationship tween nodes
Original
by algebriic identiiee new variables that equate to f read original variables into
int
int
while x + :11g0 + gl1 glll -1-gE+gl -2+g0+011
100000)
the now variables
16gl-g0 glag1> 1ag0-g13;
(igl-gl g0xg0) lighug0) (11g) x
9l g011 -1egl-e0 -2-g1-g01;
g0 198 x yer gt g0 689 -
1//8 acsert ix 211
Table Mutations for the Loop dataset on an example. For all the programs, (x = 11y11yy-0 0) 2 1 is a valid loop invariant.
31

---

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
85%
JunkHint
80%
junkNoHint
75%
readOriginal
70%
WriteOriginal
65%
Xoriginal
60%
Accuracy (%)
55%
50%
45%
40%
35%
30%
25%
20%
15%
10%
Uama-3.1-88
Uama-3.3-708
GPT-40
GPT-o1
Phi3-medium
Phi-4
Pigure 25. Evaluation various models on the Loop dataset and mutated versions.
32


Reviews:
[{"review_id": "UH16vbaQG9", "reviewer": "Reviewer_2n8z", "summary": "This work creates a framework for LLM reasoning evaluationexpands and scales up LLMs reasoning evaluation by means of an automated pipeline that converts benchmark problems into symbolic representations and then back again, and a 'mutations' step to create variations in pre-existing questions to further test reasoning capabilities.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Experimental Designs Or Analyses and Weaknesses: should directly compare against related works methods to show concrete improvements\n\nWe highlight that **the goal of Re-Imagine is to establish a unified reasoning hierarchy that integrates both previously studied mutations and the new mutations introduced in our work**. \n\nWe re-implement the two most widely studied mutations from previous work—UselessInfo and SampleValues—while scaling them up to the entire GSM8K test set. Since no prior studies have focused on evaluating models' reasoning abilities through mutations in the other three benchmarks—Loop, CruxEval, and CLadder—there is no meaningful mutations to be included in the framework.  \n\n---\n# Response to Experimental Relation To Broader Scientific Literature: Why Symbolic Representation?\n\nThe primary reason for introducing the executable symbolic representation is to obtain ground truth answers for the mutated examples in a deterministic manner. For instance, if the symbolic representation is a Python code snippet, the answer to the mutated question is derived by executing the mutated code.  As a result, both the symbolic-to-mutation and mutation-to-answer modules in the pipeline are deterministic. The other two modules—NL-to-symbolic and mutation-to-NL question—rely on LLMs. Given that LLMs have demonstrated relatively high reliability and robustness in NL-to-code and code-to-description tasks, the accuracy of these two modules is generally satisfactory. On top of these, for each LLMs-based module, we conduct verifications to further guarantee the mutations accuracy:\n* NL-to-symbolic: see lines 246-249, left column, in the main paper. \n* Symbolic-to-NL question: see line 220-224, right column, in the main paper. \n\n**In conclusion, executable symbolic representations enable us to break down the mutation process into smaller steps that are either deterministic or easily handled by LLMs, and can be thoroughly verified.** In addition, the symbolic representation gives us greater control over the mutation generation process. **We can explicitly define and study different types of mutations.** \n\nWe are unclear about which \"simpler mutation methods\" the reviewer is referring to. We assume they might be methods that generate mutated QA pairs directly using LLMs without the assistance of symbolic representations. However, as shown in the paper, LLMs struggle with answering mutated questions, and relying on them to generate ground truth answers for mutated questions can be risky. Additionally, there are no reliable validation methods to verify the accuracy of the generated mutations. We are open to further discussion if the reviewer provides more details about the simpler mutation methods they have in mind. \n\n---\n# Response to Experimental Questions For Authors: Zero-shot Only?  \n\nFor all four benchmarks, we adopt the standard in-context learning methods commonly used in the original benchmarks. In previous studies, zero-shot learning has been most widely used in CLadder, CruxEval, and Loop, while 8-shot learning has been the most common setup for GSM8K. We replicate this configuration in our experiments. In Appendix B.2, we further extend our experiments to explore the impact of different types of in-context learning examples on answering mutated GSM8K questions.  \n\n---\n# Response to repeated text in pages 2-4\n\nWe did not find any repeated paragraphs or quotations as mentioned by the reviewer (any specific line number references to this would be greatly appreciated); however, we appreciate the note and have performed an additional pass editing this section for proofreading and clarity. We acknowledge that the typesetting of the table and caption may have led to some confusion or re-reading of earlier text, and we improve the layout for the camera-ready submission.", "claims_and_evidence": "The mutation step in particular offers a great way to augment data and benchmarking for LLMs. This is a versatile tool that uses powerful symbolic reasoning tactics to test reasoning.", "methods_and_evaluation_criteria": "Because the mutation part to me is the best methodological novelty of this paper, I would personally like to see more comprehensive testing on the mutations, apart from the selective manual verification you did.", "theoretical_claims": "None", "experimental_designs_or_analyses": "The work does not do experimentation on how models perform on their framework versus other reasoning evaluation methods (like those referenced in the related works section from from Mirzadeh et al., Lewis & Mitchell, or Gonzalez & Nori). Extremely thorough model testing (Figure 2) and benchmark usage (code benchmarks).", "relation_to_broader_scientific_literature": "Generally speaking, the thing I find least convincing is the reason *why* this framework is needed overall. The mutation step is interesting, but is the symbolic conversion approach better than simpler mutation methods? This isn't particularly addressed experimentally. Of course, it makes sense logically that symbolic conversion would allow for better code/math mutations. But this can be formally shown.", "essential_references_not_discussed": "I think one aspect you didn't really cover is the adversarial nature of a lot of the testing that is done in this field. The mutations could include something like robustness testing. Also, similar works to be cited would be DreamCoder and SATNet.", "other_strengths_and_weaknesses": "Weaknesses: should directly compare against related works methods to show concrete improvements. \nStrength: very thorough testing of diverse domains, datasets, ablations (in appendix) and mutation types. Interesting findings about complex reasoning and weaknesses of these LLMs", "other_comments_or_suggestions": "I think pages 2-4 need some editing. There seems to be accidentally duplicated paragraphs (the Judea Pearl quotes, the definitions of the layers etc)", "questions_for_authors": "All prompts are done with zero-shot prompting. Did you look at all at few-shot?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xmh9mHinoR", "reviewer": "Reviewer_RCQD", "summary": "The paper mainly introduces a benchmark synthesis pipeline for math and coding reasoning problem. The proposed pipeline can make modifications of the original benchmark (question, answer) pairs to make it a different (potentially more challenging) instance. The main motivation is to evaluate the true reasoning ability of the models other than memorizing training set. The pipeline comprises 3 steps: (1) NL-to-code transformation (2) make changes to the code/computation graph (3) transform back from code to NL. The results show that models usually yield worse performance on the synthesis benchmarks.\n\n## Update after rebuttal\nOverall I think this work is a moderate extension of the Gsm-symbolic paper; I will keep my already possitive rating", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Claims And Evidence: Robustness to stochastic synthesis\n\nWe believe there may be a misunderstanding about the experimental setup. To clarify: \n\n* **All models in the paper are tested on the same generated benchmark instantiation**, ensuring fair comparison between models. \n* We also report the **statistical accuracy of models on GSM8K numerical answer predictions in Figure 7**. The bar plot presents the average accuracy and variance for each model, tested on 10 test variations per mutation, sampled using 10 different seeds. Importantly, these 10 test variations are identical across all models in this experiment. The results show the robustness of the metric to stochastic synthesis.  \n\nWe thank the reviewer for pointing out the unclear instructions on how to evaluate using dynamic benchmarks like Re-Imagine. We will add the following clarification to the paper: \n\n*For a fair comparison, all models should be tested using the same data. In practice, to evaluate using dynamic benchmarks like Re-Imagine, we recommend that researchers: (1) sample multiple test variations initially; (2) record and report the random seeds in publications or repositories; (3) report the statistical accuracy on the sampled test variations for both baseline models and proposed approaches.*\n\n# Response to Methods And Evaluation Criteria: What exact models are used for (1) NL-to-code (2) Computation graph parsing (3) code-to-NL?\n\nWe use language models only in the NL-to-code and code-to-NL steps. All mutations to symbolic representations (ie. computation graphs) are performed explicitly by modifying the AST (abstract syntax tree https://docs.python.org/3/library/ast.html) data structure of the code according to user-defineable rules.  \n\nFor NL-to-code and code-to-NL: \n\n* GSM8K \n  * NL-to-code uses Mixtral-8x7B (see line 240-244, left-column in the main paper). \n  * Code-to-NL uses GPT-4o (see line 266-270, left-column in the main paper, Figure 13 and 14 in Appendix B). \n\n* CLadder \n  * NL-to-code uses the causal engine offered in the original CLadder benchmark (see line 1124 in Appendix C). \n  * Code-to-NL uses Meta-Llama70BInstruct (see line 1140 in Appendix C). \n\nLoop and CruxEval start from the symbolic representation, so the NL-to-code and Code-to-NL steps are skipped; thus, applying Re-Imagine to these benchmarks does not require use of any models.", "claims_and_evidence": "The claim is: fixed benchmarks are not good enough as there can be leakage, so we need this benchmark systhesis tool to alter the original benchmark to get more faithful evaluation results\n\nI feel the \"sysnthesis\" part is clear and the results well supports the claim, we observe quite some drops when altering the original question, even if it is just changing values or adding irrevant information; (earlier work such as [1] also have this observation and applied similar techniques);\n\nHowever, the claim that this can be a useful \"benchmark\" is unclear; the main goal of a benchmark is to evaluate performance accross different models, however, if the synthesis is not deterministic, it is not possible to compare results with previously reported scores from other models; if we fix the synthesis, then the problem goes back to a \"fixed\" benchmark with potential leakage once released; \nIf the authors can show that even though the synthesis is not deterministic, some metric e.g., the drop of performance, can still be robust enough to have comparison across different models, the contribution of this paper would be more clear.\n\n---\nReference:\n[1] Mirzadeh, Iman, et al. \"Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models.\" arXiv preprint arXiv:2410.05229 (2024).", "methods_and_evaluation_criteria": "Some important technical details seems to be not explained (or hard to find), this includes: What exact models are used for \n\n(1) NL-to-code\n\n(2) Computation graph parsing\n\n(3) code-to-NL", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Solid", "relation_to_broader_scientific_literature": "This is related to, and can be viewed as an expansion of: Mirzadeh, Iman, et al. \"Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models.\" arXiv preprint arXiv:2410.05229 (2024).", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RCItm6OOiu", "reviewer": "Reviewer_Sq1z", "summary": "To identify whether the performance improvement of LLMs on public benchmarks such as GSM8K indeed comes from the stronger reasoning capabilities or results from mere memorization of training cases, the authors propose RE-IMAGINE to automatically make multi-level modifications to questions in the existing benchmarks. The authors employ RE-IMAGINE to generate problem variations based on 4 benchmarks and observe performance drop of LLMs on the problem variations, which is said to indicate the model reliance on recalling training data.\n\n## Update after rebuttal\n\nMost of my concerns are solved. I have raised my score from 2 to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback.\n\n---\n# Response to weakness 1: Novelty and the Influence of the Question Difficulty \n\n## (1) Novelty and contribution: \n\nIn summary, \n\n* We present the reasoning ladder for LLMs, which systematically defines different levels of reasoning difficulty. This framework establishes **a unified reasoning hierarchy that integrates both previously studied mutations and the new mutations** introduced in our work. (Please refer to our response to Q1 for further explanation of the reasoning ladder.)  \n* Alongside the reasoning hierarchy, we introduce— to the best of our knowledge—**the first scalable mutation generation pipeline that applies across multiple benchmarks and tasks**. This framework enables the creation of an arbitrary number of mutations at each level of the hierarchy for existing benchmark problems. (We elaborate on the scalability further in our response to Weakness 2.) \n\nCompared to the previous studies, \n\n* We pointed out that **previous work is primarily limited to Level-2 mutations** ([1,2,3,9]), which evaluate a model's ability to generalize beyond existing benchmarks while preserving the original reasoning path of the questions.  \n* Huyuk et al. [4] explored a level-3 mutation, Bi-Counterfactual. However, like previous studies on level-3 mutations, their approach heavily relies on manually crafted patterns and rule annotations. To the best of our knowledge, **no scalable solution has been proposed for generating problem variations across different reasoning levels spanning multiple benchmarks and tasks**. \n\n## (2) Disentangle the decline in model performance from the question difficulty: \n\nWe use GSM8K as an example. We quantitatively define the difficulty of a numerical question reasoning using the number of calculation steps in the code snippet, following iGSM [2]. We compute the average accuracy of each model across examples with varying numbers of calculation steps. Given the large number of models tested, we aggregate their results and present the overall average accuracy in the following table. We included detailed results for each model respectively in the Appendix of the paper. \n\n| Intervention Type | 2 steps | 3 steps | 4 steps | 5 steps | 6 steps |\n|-------------------|------|------|------|------|------|\n| Raw | 0.95 | 0.94 | 0.84 | 0.91 | 0.83 |\n| SampleValues | 0.87 | 0.84 | 0.75 | 0.74 | 0.80 |\n| UselessInfo | 0.91 | 0.90 | 0.90 | 0.81 | 0.88 |\n| CounterFactual | 0.74 | 0.71 | 0.75 | 0.62 | 0.67 |\n| InsertConditional | 0.62 | 0.68 | 0.65 | 0.61 | 0.57 |\n| AddDependence | 0.57 | 0.47 | 0.46 | 0.45 | 0.42 |\n\nFrom the table we make three key observations: \n\n* In nearly all scenarios, **even when tested on examples with the same number of calculation steps, models consistently perform worse on the mutated sets compared to the original test set**. \n* Compared to Level-2, Level-3 mutations present a significantly greater challenge. Especially, **the accuracy on Level-3 mutations with just two calculation steps is lower than on Raw test examples with six calculation steps with a significant margin.** \n\n---\n# Response to weakness 2: Generalization of the method\n\nThe pipeline requires three types of adapters: Question-to-Symbolic Adapters, Symbolic Representation-to-Mutation Adapters, and Mutation-to-Natural Language Question Adapters. The pipeline is designed in a **modular plug-and-play fashion**, making **all adapters both reusable and customizable**. Users can either utilize existing adapters if they meet their needs or modify them to suit their specific dataset or domain. The estimated manual effort required for **adapter customization** is based on the four domains we implemented: \n\n* Question to Symbolic Adaptor: **Prompts Writing**.  \n* Symbolic Representation to Mutation Adapter: **Write around 50 lines of code** to define the mutation of the symbolic representation.   \n* Mutation to Natual Language Question Adaptor: **Write around 100 lines of code for model prompting** to translate the mutation to natural language. \n\n**We replaced the word \"automatic\" with the more precise term \"scalable\" in the paper.** \n\n---\n# How do the three levels relate to the causal ladder? \n\nThe key connection between Pearl's ladder and our framework is the problem's computation graph, which can be understood as a causal model in Pearl's framework. Each problem in a benchmark can be interpreted as a single realization of the graph with specific node values. Experiment associated with different perturbations in such graph can be related to operations in Pearl's ladder of causation. For instance, computing the effect in the outcome of the change in one leave node maps to the definition of a counterfactual. Note, however, that that not all mutations in the three levels have a causal counterpart (like adding an irrelevant piece of information or changing an operation). In this sense our framework can cover a broader definition of reasoning in each level.", "claims_and_evidence": "The paper makes two primary claims: 1) RE-IMAGINE can automatically generate multi-level problem variants, thereby extending existing datasets and mitigating data leakage during training; 2) The performance gap between raw questions and their variants suggests that LLMs rely on recalling training data.\n\n* **Evidence for claim 1:** The authors claim that RE-IMAGINE is a general framework and an automated pipeline. But how to operate the first step, language-to-symbolic transformation, is task-specific and requires manual intervention. And whether or why the mutation mentioned in the paper is general and can be applied to other domain is not clear. The mutation seems to need humans to manually define and restrict the modification types. For example, for the \"Sample Values\", if the values are numbers, boolean, or strings, the mutation may be different. The authors should provide more details on the generalization of the proposed method and how much manual work is necessary to adopt it in a new dataset. If this framework requires manual redesign of step 1 and step 2 for each new dataset, I think the applicability of this framework is insufficient.\n\n* **Evidence for claim 2:** I doubt whether the experimental results sufficiently support claim 2.\n\n  * Most modifications introduced by RE-IMAGINE significantly increase the complexity of the problems. The UselessInfo operation at level-2 introduces an extra node in the computation graph, and level-3 modifications add reasoning steps. Consequently, the observed decline in model performance could be attributed to either the increased difficulty of the problems or the model's reliance on memorized training data and poor generalization. Previous studies, such as [1] and [2], have shown respectively that irrelevant information and additional reasoning steps can negatively impact model performance.\n\n  * The SampleValues modification at level-2, which involves integer and float values fluctuating within the range of [-10, 10], preserves the problem's difficulty. This suggests that the performance drop in this case may indeed reflect the model's dependence on training data for reasoning. However, this conclusion is not novel, as [3] has already demonstrated in GSM8K that LLMs are sensitive to minor changes in variable values and names, leading to performance degradation.\n\n  * So to robustly support claim 2, the authors need to disentangle the effects of increased problem difficulty from the model's reliance on training data. Without this clarification, the claim remains partially unsubstantiated.", "methods_and_evaluation_criteria": "RE-IMAGINE supports automated data generation through a pipeline involving: language-to-code transformation, mutations of symbolic representations and code-to-language transformation. The idea is simple and clear. However, the authors should provide more details on the generalization of the proposed method and how much manual work is necessary to adopt it in a new dataset.\n\nThe evaluation metric includes model accuracy towards raw questions and the question variations. Besides, in Sec 4.3, the authors also involve the metric of sufficiency / necessity inconsistency, which is proposed in [4]. I recommend that the authors provide a more detailed explanation of the experimental setup and the methodology for this metric in the paper. Currently, the paper merely references [4], which may confuse readers unfamiliar with the prior work.", "theoretical_claims": "The paper introduces a hierarchical framework to evaluate the reasoning ability of models. While this framework is inspired by the hierarchical structure proposed by Pearl in the context of causality, I find the connection between this framework and causality to be unclear. In other words, could you justify why this particular framework was chosen or how it is fundamentally linked to causality?", "experimental_designs_or_analyses": "See Claims and Evidence, ablations for the effects of mutations on question difficulties are needed.", "relation_to_broader_scientific_literature": "The paper discusses a crucial problem concerning the evaluation of LLM reasoning capabilities, which is how to identify genuine reasoning and merely recalling training data. This has long been a hotly-debated question in the community of LLM reasoning. Here are some literatures recommended for reference under the topic:\n* Reasoning Elicitation in Language Models\nvia Counterfactual Feedback, Hüyük et al., 2024, https://arxiv.org/abs/2410.03767\n* Counterfactual Memorization in Neural Language Models, Zhang et al., 2023, https://arxiv.org/abs/2112.12938.\n* Faith and Fate: Limits of Transformers on Compositionality, Dziri et al., 2023, https://arxiv.org/abs/2305.18654\n* Case-Based or Rule-Based: How Do Transformers Do the Math?, Hu et al., 2024, https://arxiv.org/abs/2402.17709\n* Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models, Hou et al., 2023, https://arxiv.org/abs/2310.14491", "essential_references_not_discussed": "The related work section is missing in the paper, and my primary concern is that the unique contribution of this work beyond the following previous work is unclear. While the paper focuses on automatically modifying public datasets, with GSM8K being a central dataset, several highly relevant studies are not discussed or cited. These include:\n\n* GSM-IC [1] shows that irrelevant context can degrade LLM performance.\n* iGSM [2] introduces a synthetic pipeline which captures parameter dependencies in a symbolic structure.\n* GSM-HARD [9] modify parameter values to be much bigger than in the original dataset.\n\nThe contributions beyond the above work and GSM-Symbolic [3] should be more clearly discussed.", "other_strengths_and_weaknesses": "Strengths:\n* The paper discusses a crucial problem of how to identify memorization and genuine reasoning in LLMs.\n* The presentation of the work is very clear and well-structured, making it accessible to readers.\n* Several modifications introduced in RE-IMAGINE, particularly at level-3, appear to be novel to the best of my knowledge. These modifications have the potential to increase the difficulty of the dataset and mitigate the impact of data leakage.\n\nWeaknesses：\n* My primary concern revolves around (1) the novelty of the proposed methods.\n  * The techniques at level-2 have been previously explored in related works [1, 2, 3].\n  * While some methods at level-3 are indeed novel, they do not maintain the same level of problem difficulty as the original tasks. As a result, the observed decline in LLM performance cannot be solely attributed to memorization, as suggested in the 'Claims and Evidence' section. This limits the strength of the conclusions drawn regarding the reliance of LLMs on memorization for answering questions.\n* And (2) the generalization of the proposed method. (See the first point in 'Claims and Evidence')", "other_comments_or_suggestions": "Here are some typos:\n1. line 019 right: \"Traditionally, the **evaluation** of reasoning **evaluation** in LLMs\" -> \"evaluation of reasoning abilities\"\n2. line 078 right: \"The proposed hierarchy has three levels of increasingly difficulty\" -> \"increasing difficulty\" ?", "questions_for_authors": "1. Please explain about how the pipeline can work accross domains and how much human efforts are needed to adopt this pipeline on different datasets. \n2. Please explain more about the cause of performance decreasing. Whether the performance of the model has declined comes from the difficulty of the problem itself, rather than the difference between \"memory\" and \"reasoning\".\n3. Please explain the novelty of these mutation and your overall framework. \n4. How do the three level are related to the three level in causal ladder?\n\nThe overall reference:\n\n[1] Large Language Models Can Be Easily Distracted by Irrelevant Context, Shi et al., 2023, https://arxiv.org/abs/2302.00093\n\n[2] Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process, Ye et al., 2024, https://arxiv.org/abs/2407.20311\n\n[3] GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models, Mirzadeh et al., 2024, https://arxiv.org/pdf/2410.05229\n\n[4] Reasoning Elicitation in Language Models\nvia Counterfactual Feedback, Hüyük et al., 2024, https://arxiv.org/abs/2410.03767\n\n[5] Counterfactual Memorization in Neural Language Models, Zhang et al., 2023, https://arxiv.org/abs/2112.12938.\n\n[6] Faith and Fate: Limits of Transformers on Compositionality, Dziri et al., 2023, https://arxiv.org/abs/2305.18654\n\n[7] Case-Based or Rule-Based: How Do Transformers Do the Math?, Hu et al., 2024, https://arxiv.org/abs/2402.17709\n\n[8] Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models, Hou et al., 2023, https://arxiv.org/abs/2310.14491\n\n[9] PAL: Program-aided Language Models, Gao et al., 2023, https://arxiv.org/abs/2211.10435", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R46BzrMmsZ", "reviewer": "Reviewer_b1Wv", "summary": "This paper introduces R E -I MAGINE: a framework to characterize a hierarchy of reasoning ability in LLMs, alongside an automated pipeline to generate problem variations across all the levels of the hierarchy. By altering problems in an intermediate symbolic representation, RE-IMAGINE generates arbitrarily many problems that are not solvable using memorization alone.  Reductions in performance can be observed when the models are queried with problem variations.\n\n## update after rebuttal\nI keep my score positive for this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Claims And Evidence: Some datasets also have a systematic hierarchy such as GPQA\n\nThe hierarchy in GPQA represents the difficulty of the **original problems**, whereas the hierarchy we introduce in Re-Imagine defines reasoning complexity through **variations of problems** from existing, well-established benchmarks.\n\nWe believe that Level-3 mutations align with counterfactual reasoning based on Pearl’s ladder of causation—**how our belief about the occurrence of event Y changes if event X had value x’ instead of x**. In Figure 1, for example, we present: \"Janet bakes muffins with 2 eggs... Assume Janet no longer bakes muffins, how much money does she make every day?\" Here, the observed outcome (event Y) is Janet’s daily earnings, while the causal factor (event X) is whether she bakes muffins or not. The logical facts are structured as a math problem, and Level-3 mutations modify one of these facts by introducing an additional assumption.\n\nWe agree with the reviewer that creating scenarios unlikely to occur in the real world, such as \"Assume Janet bakes muffins using 10,000 eggs every day,\" can be highly interesting. Thanks to the modular plug-and-play design of our pipeline, we encourage the community to implement mutations of their interest to thoroughly assess models' reasoning abilities.\n\n---\n# Response to Theoretical Claims 2: NL-symbolic-NL process\n\nWe recognize that the accuracy of NL-symbolic-NL translation is critical. Therefore, for benchmarks requiring full NL-symbolic-NL translation, such as GSM8K, we employ four methods to ensure the correctness of the mutated QA pairs and report results with quantified noise:\n- **NL-symbolic:** We select Python solutions that not only produce the correct answer but also ensure that all constant variables in the code (root nodes in the computational graph) align with the numbers in the question, and vice versa (see lines 246-249, left column, in the main paper).\n- **Symbolic-NL:** to ensure the accuracy of the symbolic-to-NL translation, we prompt GPT-4o a second time to back-translate the mutated math problem into Python by modifying the original question’s Python solution. The generated code must produce an execution result that matches the ground truth answer of the mutated question (see line 220-224, right column, in the main paper).\n- **Manual Quality Check:** we manually check the quality of the mutated QA pairs and report the error rate in each mutation (see line 227-235, right column, in the main paper).\n- **Report Results with Quantified Noise:** we show the error rate of the mutated data when reporting the model's performance (see Figure 2). \n\n---\n# Response to Experimental Designs Or Analyses 2: Error analysis of the relationship of questions across levels, and training on level-3 questions.\n\nWe emphasize that this study primarily focuses on introducing the reasoning hierarchy, the benchmark mutation pipeline, and evaluating models' zero-shot and few-shot performance on dataset mutations. Model training is beyond the scope of this paper. However, thanks to the scalability of our proposed pipeline, we are expanding training set mutations to investigate whether exposing models to mutated questions during training can enhance their reasoning abilities. However, the results will not be included in this paper.\n\nIn Appendix B.2, we present experiments in an in-context learning setting, revealing that (1) simply replacing the original demonstrations in the context with mutated ones does not improve models’ reasoning accuracy. However, (2) models perform significantly better on generated test set variations when provided with both original and mutated examples as demonstrations. These findings offer insights for future model-training experiments.\n\n---\n# Response to Experimental Designs Or Analyses 3: More popular LLMs should be adopted for analysis, such as Qwen\n\nWe conduct additional experiments with a broader range of popular LLMs, and the observations in the paper still hold for these new models. \n\n**Loop** (mutations Raw, JunkHint, JunkNoHint, readOriginal, WriteOriginal, Xoriginal)\n\n- QwQ-32B:\t75.9%\t76.7%\t66.5%\t63.67%\t39.59%\t36.33%\n- R1-Distill-Llama-70B:\t75.10%\t62.45%\t49.80%\t60.41%\t49.39%\t48.57%\n\n**CruxEval** (mutations Raw, Mutate String (L2), Mutate Value (L2), Redefine Function (L2), Replace Operator (L2), Swap Conditional (L2))\n\n- GPT-4.5:\t45.5%\t23.55%\t32.45%\t29.79%\t29.14%\t32.06%\n\n- GPT-o3-mini:\t56.88%\t35.95%\t59.82%\t56.35%\t58.52%\t50%\n\n**GSM8K** (mutations Raw, SampleValues, OverWriteValue, UselessInfo, AddDependence, InsertConditional)\n\n- qwq-32b 100  98.2  92.6 99.1 59.6 95.6\n\n- r1-distiall-qwen-32b 98.3 93.5  85.9 97.6   63.2  85.0\n\n- GPT-o3-mini 97.4 90.3  84.02   93.5 77.3  91.6\n\n- GPT-4.5 97.45  89.76 81.28 95.47 61.54 89.28\n\nAdditional experiments are ongoing. We will include performance results for qwq, r1-distill-qwen, GPT-o3-mini, and GPT-4.5 across all four benchmarks in the camera-ready, and an error analysis in the Appendix.", "claims_and_evidence": "Not for all.\nProblematic claim: these variations have been developed in an ad hoc manner, lacking a systematic hierarchy.\nSome datasets also have a systematic hierarchy such as GPQA.", "methods_and_evaluation_criteria": "It makes sense.", "theoretical_claims": "1. the use  of theory on \"counterfactual\" (for generating level 3 questions)\nA counterfactual is a hypothetical scenario that describes what would have happened if a different decision or condition had occurred, contrary to what actually happened in real-world. While in the existing questions, they do not present real-world questions, making the questions of level-3 more like intervention. Maybe creating an scenario which would never happend in real-world could alleviate this issue.\n\n2. The validity of the method might be influenced by the correctness of the NL-symbolic-NL process.\nA wrong python program can also produce the same answer as the ground truth.", "experimental_designs_or_analyses": "1. the evaluation on the created benchmarks on various domains is somewhat sound.\n\n2. More analyses are needed.\nAll experiments are about evaluation on the created benchmark (too much content is used for this part). Some analyses should be incorprate to investigate the relationship of question in different level. For example, whether training on level-3 questions can help answering level-2 questions.\n\nMore popular LLMs should be adopted for analysis, such as Qwen.\n\nError studies should be done to analyse why LLMs perform worse on hader questions.\n\nCase studies are also required.", "relation_to_broader_scientific_literature": "It might influence the evaluation of LLMs.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "Please refer to the weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper gives a framework to synthesize benchmarks for coding and math reasoning tasks. The framework involves: (i) converting natural language problem statements to a code representation, (ii) applying mutations to this representation, and (iii) transforming the problems back from code to natural language. The approach is used to evaluate several popular LLMs. It is shown that models usually perform worse on the mutated benchmark. The conclusion is that LLMs often recall on statistical recall when performing "reasoning" tasks. 

The reviewers generally agree that the approach is reasonable and the claims are sound. (A comparison with direct mutations to natural language problem statements would have made the paper stronger but this is not a major weakness.) The approach isn't particularly surprising, and previous work has studied ideas closely related to the paper's. However, the paper deserves credit for connecting these ideas into a unified framework, and the connection to Pearl's causal ladder is also appealing. I would put this paper in the "accept if there's room" category.
Decision: Accept (poster)