MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofs
Authors: Andreas Opedal
Haruki Shirakami
Bernhard Schölkopf
Abulhair Saparov
Mrinmaya Sachan
Venue: ICLR 2025 Poster (2025)
Keywords: Arithmetic reasoning
evaluation
proofs
large language models
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a framework for evaluating language models on math word problems with proof trees of arbitrary complexity

Abstract: Large language models (LLMs) can solve arithmetic word problems with high accuracy, but little is known about how well they generalize to more complex problems. This is difficult to study, as (i) much of the available evaluation data has already been seen by the most capable models during training, and (ii) existing benchmarks do not capture how problem proofs may be arbitrarily complex in various ways. In this paper, we present a data-generation framework for evaluating LLMs on problems with arbitrarily complex arithmetic proofs, called MathGAP. MathGAP generates problem statements and chain-of-thought reasoning traces according to specifications about their arithmetic proof structure, enabling systematic studies on easy-to-hard generalization with respect to complexity of proof trees. Using MathGAP, we find that LLMs show a significant decrease in performance as proofs get deeper and wider. This effect is more pronounced in complex, nonlinear proof structures, which are challenging even for the most capable models. The models are also sensitive to simple changes in sentence ordering. However, they remain capable of solving some complex problems, suggesting that reasoning generalization is noisy.

Published as a conference paper at ICLR 2025
MATHGAP: OUT-OF-DISTRIBUTION EVALUATION ON PROBLEMS WITH ARBITRARILY COMPLEX PROOFS
Andreas Opedal 1.2, Haruki Shirakami'.5 -3,.
Bernhard Schölkopf
Abulhair Saparov
Mrinmaya Sachan'
'ETH Zürich Max Planck Institute for Intelligent Systems, Tübingen
Idiap Research Institute
*Purdue University
andreas opedal 1 @iinf.inhe ethz.ch
haruki. shi shimakami @idiap.cd...
ABSTRACT
Large language models (LLMs) can solve arithmetic word problems with high accu- racy, but little is known about how well they generalize to more complex problems. This is difficult to study, as (i) much of the available evaluation data has already been seen by the most capable models during training, and (ii) existing benchmarks do not capture how problem proofs may be arbitrarily complex in various ways. In this paper, we present data-generation framework for evaluating LLMs on problems with arbitrarily complex arithmetic proofs, called MathGAP. MathGAP generates problem statements and chain-of chain-of-thought reasoning traces according to specifications about their arithmetic proof structure, enabling systematic studies on easy-to-hard generalization with respect to complexity proof trees. Using Math- GAP, we find that LLMs show significant decrease in performance proofs get deeper and wider. This effect is more pronounced in complex, nonlinear proof struc- tures, which are challenging even for the most capable models. The models are also sensitive to simple changes in sentence ordering. However, they remain capable solving some complex problems, suggesting that reasoning generalization is noisy.
INTRODUCTION
High performance on reasoning benchmarks often taken as evidence that transformer based large language models (LLMs) can "reason". However, many current benchmarks unreliable, it is likely that the problems they contain are present in the model training data (Sainz et al. 2023 Deng et al. 2024 /Zhang al. 2024). Moreover, most evaluations fail to capture that reasoning problems can be arbitrarily complex through composition of subproofs and use of multiple rules of inference. High accuracy on specific set of problems in a benchmark dataset therefore not sufficient to conclude that LLMs can generalize to more complex, unseen problems. To obtain a more appropriate empirical measure of reasoning ability, one must evaluate on data that not present in any benchmark dataset, containing proofs that are more complex than the ones the model has already seen.
In this paper, we aim satisfy these desiderata. We consider math word problems (MWPs), which are one of the most frequently used testbeds for L.M reasoning, and propose framework for evaluating Mathematical Generalization on Arithmetic Proofs MatiisAP.. We start by discussing how solutions to MWPs can be characterized in terms of proof trees, in which nodes are labeled with logical statements about the problem under a world-model framework (Opedal et al., (023). and the edges are induced by proof steps such logical statements. Doing so reveals several ways of characterizing a problem's 's complexity based on its proof tree, e.g., depth, width, shape, and the ordering of nodes (all defined in $3.)). With this formalism, we develop method to generate problems with specified proof tree characteristics. Because this method is based on proof trees, it also enables rich annotations in the form of ground-truth reasoning traces. More specifically, traversing the proof trees in post order, we obtain chain-of-thought (CoT, Wei et al.) (022)) reasoning traces in an automated manner. Fig. Iillstraaes the generation method, showing a proof tree having depth width 5, what we call nonlinear shape, and where the ordering of the sentences is given by traversing the leaf nodes in left-to-right order.
"qqual contribution.

---

Published as a conference paper at ICLR 2025
Step 1: Givan a logical form, sample proof tree by heratively appyynng inference untila a stopping oritation has been reached.
Problemt specification:
Available logical form templates:
Avuilable inference rulcs:
Nonlinear
1. cent [agent), Equassiit tyll Jessityll
ooen compl cont
D
2. coep [agenttl] (sgentz). [gpantiity lecsityll
coaci.... const...
coep I...)
Width:
3. coep eet (mgent3], (egent)). (agena3) (agenta) (entityl)
compl fooel comp eqg conti....)
Canonical ordering
[...]
(I, 17,, apple))
comp ap.ppl)))
cont apple)
cont 19, applpp))
con't (2,, 2?,pppl
comp (E, applle)
comp- apple)
cont 18,19,,ppi0.
Step 2:
Create a word problem by mapping leaf nodes to text body and root node to question using tamplates.
Step 3:
Generata solution by mapping of steps, Internal nodes map CoT explonations and root node to snswer.
mone apples tham Issbuila, 4 Juhn has apples @ Emily 19 apples. The number of apples that Lacy has more than Sam is the sume as the difference between the number of apples that John has componed to Emily.
lesbellas 17 apples. Lury
17 apples. Lucy more apples Isabella. apples. Emily
@ So has 20-27 apples. John
applex. So dilleenenc butwaem number of apples John and Emily have The number of apples that Lucy has more then Sam the same as the difference betwees the number of apples that John has compprred to Emily. So Sam has 27 27-8-18 apples.
many apples
have?
We propose MathGAP, an evaluation framework for arithmetic reasoning in which LLMs are tested on problems with proofs of arbitrary complexity. This diagram shows how problems and CoT solution annotations are generated under formalism. The complete list of logical forms and inference rules that we consider in our experiments are given in Tables I and 2 respectively.
Figure I:
This generation method forms the backbone of MathGAP, our framework for studying out-of- distribution (OOD) generalization gaps on arithmetic proofs. MathGAP avoids data contamination since the distribution of generated test problems can always be different from those in training or in context, by design. When the performance on problems at one level of complexity hits saturation, we can flexibly generate a new set of problems that are even more complex. This is in spirit similar to Dynabench (Kiela et al. 2021) but, importantly, does not require human annotators.
MathGAP enables many new empirical investigations. In this paper, we perform a systematic analysis on whether LLMs can use simple examples in context to solve more complex ones at inference. To this end, we consider two different distributions of in-context CoT demonstrations: examples with only the simplest possible proof trees, and range of examples with varying complexity, akin to curriculum learning (Bengio et al. 2009). We then evaluate on test sets containing problems that are increasingly more complex than the in-context examples. The performance is further compared to in-distribution and zero-shot baselines. Our study includes Mixtral-8x7B (Jiang et al. 2024a). Llama3- 8B, Llama3-70B (Llama Team 2024), GPT-3.5-Turbo, GPT-4o (OpenAl 2024), and a few additional evaluations on ol -preview and DeepSeek-RI (DeepSeek-Al] 2025). We present several findings:
i) Generalization proof width is harder than to proof depth, but there is a steady decrease in per- formance as proofs get both deeper and wider. The performance decrease is particularly notable for the so-called nonlinear problems; solving them requires keeping intermediate steps longer in memory than for linear problems, for which intermediate steps can be consumed immediately (under post-order traversal). Even GPT-4o fails solve the most complex nonlinear problems.
ii) However, the performance decrease is moderate in most cases. This suggests that the models are able to generalize to some extent, albeit with a noisy form of reasoning (Prabhakar et al. 2024). iii) All models are sensitive to a simple change in sentence ordering, in which one sentence is moved to the front of the problem. Moreover, we find that the problem is casiest for LLMs if the sentence is moved from the beginning the end rather than from the middle of the problem. This is surprising since one would expect a monotonically decreasing relationship between accuracy and the distance of movement.
iv)
We do not see a consistent advantage to using in-context examples. Contrary to our expectations, in-distribution examples are not always beneficial for performance, as compared to zero-shot. This suggests that some problems are hard to learn even when the LLM is given demonstrations of similar problems, independently of its generalization ability. Demonstrating range of examples with varying complexity is usually preferable to demonstrating simple examples, suggesting that LLMs benefit from diverse prompt.
v) ol and R1 outperform the others on nonlinear problems. However, we are able to create test set on which they solve only 5% and 1 11% respectively-indicatnn that MathGAP is future-proof.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Data contamination. Machine learning. broadly, is the study of algorithms that can learn from data and generalize to unseen data. Machine learning models must therefore be evaluated unseen test sets. As it turns out, modern-day LLMs are not being properly evaluated in this regard. Much of the data they are evaluated on has already been used during model training, both labeled (Dodge et al.) 2021, Deng et al., (2024) and unlabeled (Elazar et al. (022)). Researchers have raised serious concerns about such data contamination issues (Jacovi et al.] [023] Sainz et al., 2023). For arithmetic reasoning specifically, Zhang et al. (2024) recently found evidence that the widely used GSM8K benchmark (Cobbe et al. (021] is contaminated and presented evaluations on a new dataset that is not publicly released. MathGAP mitigates data contamination by creating new, more complex synthetic test sets.
Generalization on reasoning tasks.
Generalization to data from different distribution (i.e., OOD data) is a core problem in machine learning and has also been studied extensively in the context
of LLM reasoning (Schwarzschild et al. 202T |An et al. [2023] Kudo et al. 2023 Liu et al., 2023
Saparov & He. 2023 Zhang et al. [2023] Mészáros et al. 2024 Thomm et al.) 2024). While LLMs
performance on OOD data can be improved by various techniques (Amil et al. 2022 BBrazjanizzadh
problems of arbitrary complexity under set of known inference rules is still an open problem. For instance, transformer-based L.MMs struggle to generalize to longer problem proofs when solving logical reasoning tasks (Dziri et al. 2023
& Piantadosi 2024 Hu et al., 2024). generalization
Saparov et al. 2023). We contribute to this literature by providing an evaluation framework for OOD generalization to complex arithmetic proofs. Our work relates scalable oversight (Bowman et al.) 2022) and easy-to-hard generalization (Burns et al., 2023 Hase et al. 2024 Sun et al. (2024). which can be studied in a principled manner with MathGAP by characterizing what is easy and what is hard.
Evaluation on MWP benchmarks. MWPs are often treated as testbed for studying reasoning
abilities of LLMs (Patel et al., [202T] Fu et al., 2023) Shakarian al.] [202]] [Stolfo et al.] 2023] Zong & Krishnamachari 2023 Huang et al. 2024 Yu et al., 2024 Ye et al. 2025); such problems are interesting because they require several distinct skills to solve, while remaining conceptually simple (Stem 1993). However, plexity in this domain. Our study in / related to Hase et al. (2024). who fine-tune LLMs on easy problems and evaluate them on harder ones inference, using GSM8K and other possibly contami- nated datasets. Their complexity metrics focus on problem length, but do not capture how a reasoning problem gets more complex (i.e., through the characteristics of the proof tree). MMizzdehh al. (2025) recently performed evaluations on a version of GSM8K in which variable names and numbers have been altered, and found that LLMs are surprisingly sensitive to such substitutions. MathGAP is more general: while it allows for such substitutions as well, our main goal is to vary the proof structures.
not aware of much work generalization in regard to problem com-
3 A FORMAL TREATMENT OF MATH WORD PROBLEMS
We aim to generate new problems of arbitrary complexity, so we require a formalism to characterize complexity in precise manner. First, we explain how the semanties expressed in MWPs can be written as sequences of logical forms ( 33.1 We then show how to apply inference rules on these logical forms to deduce new ones (3.2)) This leads to view of problem solutions as proof trees, the structure of which can be used to characterize the reasoning required to solve the problem.
3.1 MATH WORD PROBLEMS AS LOGICAL FORMS
Each MWP is represented as sequence of logical forms under the formalism from Opedal al.) (2023, 2024). which is a subset of first-order logic. A logical form is a truth statement about the world that is being described by the problem, representing some arithmetic relationship between the number of entities possessed by one or several agents. Such statements can refer to the number of entities an agent has, how many more entities an agent has relative to another, the action of an agent giving their entities to another agent, etc. Formally, logical form represents the semantics of a single sentence and consists of a predicate that takes a set of properties as arguments. Every logical form has agent property, an entity property, and quantity property. Other, optional properties include unit and attribute. We define a model as a sequence of logical forms that align with the sentences of problem text, thus representing the semantics of the whole problem.
3

---

Published as a conference paper at ICLR 2025
Table 1: Example logical forms. A logical form comprises a predicate and a set of properties specific to that predicate. Each logical form induces natural language sentences that preserve its semantics.
Logical Form
Example Templates
Example Sentences
Predicate (abbr.)
Properties
container
(agentma,
(cont)
quantity=q. entityme. attributesk, unitmu]
taj hos (95 191s (KJ yejs.
Alice has kilograms of red apples.
[a] owns (q) [f]x (k) [o]x.
Alice owns 5 kilograms of red apples.
comparison (cong)
(agentA=a. agentB-b,
[b] hos [q] fewer [ejs thon 14).
Bob hos fewer apples than Alice.
quantity=g.
[a] has [q] [e]s than [bb].
Alice has apples than Bob.
entity=e)
transter
(B
[a] gave [b] [5] [o]x.
Alice guve Bob 3 apples.
serder_agent
quantity=q,
[b] got [q] [e]sfrom [a].
Bob got apples from Allce.
entity=e)
partwhole
(thollo.ogon==:A..p.
[arh and [o.,] have (q) [o]x combined.
quantitymg.
Alice and Bob have 8 fruits combined.
whole_encity=e)
comp (agentA=a,
The mumber of [e]s that [c] has more equal the difference between the number of Le that ial and [b] have.
The number of apples that Charlle has more than David equat difference berseen the number of apples Alice and Bob have.
ageatB=b, agentC=e,
ageatD=d. entity-ee
For a given predicate and set of properties, we write predi cate (propertyl=x, property2=y, ...) to express a logical form. Property names are, however, often omit- ted for brevity, as in predicate (x, Y, ... There are two types of predicates: (i) those that represent the ownership of some entity by an agent, called cont a ainer (abbreviated cont), and (ii) those that represent arithmetic relationships between two sets of properties. The relationship predicates correspond to arithmetic concepts and are based on taxonomy from the learning sciences (Riley et al. 1983); we use transfer. compart sin son (abbreviated comp), part who le and comp-eq The comp-eq predicate was introduced in this paper in order to construct nonlinear problems of arbitrary length-see App. D for details. Table [ gives concrete example sentences for logical forms with each of these predicates, Fig. shows the world model representation of the problem in Fig. 1 and App. provides further intuition
The logical forms in the world model of problem separated into body and question. The question is a single logical form that appears at the end of the world model. It holds placeholder variable instead of an explicit quantity, which represents the correct answer to the problem. For example, the interrogative sentence "How many apples does Alice have?" is represented by the question cont (Al ice, q apples le), where q is placeholder variable. The body consists of all logical forms in the world model that are not questions. The answer problem is the logical form declared by the question with the correct numerical quantity substituted into its placeholder variable.
3.2 CHAIN-OF-THOUCE SOLUTIONS AS PROOF TREES
Solving MWPs is form of deductive reasoning. where the correct answer follows from what is stated in the text through combination of rules of arithmetic and world knowledge. We use such derivations, called proofs, to characterize these problems. The proofs in our setup are analogous to
those in other proof systems, such as natural deduction (Gentzen, 1935 Pfenning 2004
We want formally reason over the logical forms introduced in $3.1 To that end we introduce set of inference rules, which are used to prove new logical forms from previously known ones. A proof step, written in Gentzen-styy notation as
L
L
LN
L
'The transfer and phrt-whole concepts are usually called change and combine, respectively, learning science literature (Nesher et al., 1982).
II order denote that the items of several agents combined under pa partwho"s e, treat the conjunction agents A Aー = a as another agent Table [ The same done for entities, as shown Table In fact, each of our inference rules shown in Tahle 2 can be decomposed into the primitive inference rules of natural deduction, using first-order logic expressions. App. demonstrates example how this is done.
4

---

Published as a conference paper at ICLR 2025
Table 2: Inference rule templates (left) with corresponding examples of rule applications in natural language (right). The variables a, b, C, refer to agents, q, 91, q2 refer to quantities, and e, refer to entities. Attribute and unit properties are excluded here, but analogous rules exist in which they are present. All inference rules are commutative except the t transfez rule; see footnote I Rule applications of an inference rule with premises L1, L2..... LN and consequent L are written as 11-112 L1 L'., where L' a natural language expression for the logical form L. Axioms are formed from the facts stated in the problem, illustrated in Fig. !
Inference Rules
Example Sentences
cont (a, e) comp (b, 9, e) cont (b, qi + qa, e)
Alice has 3 apples. Bob has 2 more apples thom Alice. H- Bob has 5 apples.
cont (a,, 9he trensser (a, b, Cav a)
Alice has 3 apples. Rob gave 2 apples to Alice. |- Alice has 5 apples.
qi qa,
cent la, Rir e) con= (b, 42 el
Alice has 3 apples. Bob has 5 appies. I- Bob has 2 more apples than Alice.
comp (b, a, 92 Ti e)
cont (81, Siy ei)
cont (der Bay ex)
Alice has apples. Bob has 5 apples. I- Alice and Bob have 8 fruits combined.
( Ei. A2a101
Alice has 7 apples. David has 2 more apples than Charlie. The number of apples that Bob has more than Alice is the some difference between number upples that apples.
cont la, Di m)
conp Id, Gr Ber comp-ng a, at woni Ba
David Charlie have.
is an instance of an inference rule where the conclusion logical form L is deduced from premises L1, L2,..., Ln- An axiom is a proof step without premises. The axioms for problem are the logical forms in the body of its world model, i.e., all logical forms excluding the question. Table 2 lists the inference rules that we use, along with examples of how they may be expressed in natural language 14 For example, consider the two logical forms cont (Tsabell l la, 17, apple) and comp (Lacy,, Isabella, 10, apple) from Fig. [ representing the facts that Isabella has 17 apples and Lucy has 10 more apples than Isabella, respectively. The proof step
cont (Isabella, 17, apple) comp (Lucy, Isabella, 10, apple) cont (Lucy, 17 10, apple)
lets us deduce the logical form cont (Lucy, 27, apple), i... that Lucy has 27 apples.
math word problem can be solved by applying inference rules until the answer to the problem has been proved. Formally, proof tree is rooted ordered tree where each node is labeled with unique logical form L, which is the conclusion of proof step with premises Iı,..., Ln matching the labels of the child nodes in the same order. Leaf nodes are thus labeled with axioms. proof of a particular problem is a proof tree where the labels on the leaf nodes are the logical forms in the body of the problem, and the label the root is the answer to the problem. The tree shown in Fig. illustrates proof of the problem given in the same figure.
Complexity of proof trees. We seek a systematic way to analyze whether LLMs can generalize from simple to more complex proofs The definition of proof given above gives rise to several ways of char- acterizing reasoning, out of which we consider four: depth, width, shape, and ordering. The depth of a proof tree is defined as its height, i.e., the maximum path length from the root node to any leaf node. Next, the width of proof tree is the number of leaf nodes, or axioms, that contains. The proof tree in Fig. thus has depth and width 5. In words, this means that the answer is two proof steps "away"
*The list is not exhaustive as additional variations on the rules exist. In particular, the logical forms may contain attributes and units (such the cont examples in Table the cont premises may be replaced by the pa partwhole premises (so that partwhole can be used for further proof steps), and the premises can be given in any order for all inference rules (i.e., they are commutative) except the one using t transfes The transfer rule is non- conmmmmmutaaive since it involves a notion of time. For instance, "Alice (now) has 5 apples. imply the same conclusion Moreover, note that the third inference rule listed in Table is commutative, since swapping the two premises yields the conclusion comp (a, b, qi 92, e), which is equivalent to comp (b, a, q2 91 e).
Alice ate 2 apples." does
"Alice 2 apples. Alice (now) has 5 apples."
$ Note that our use of the term "complexity" is different from its conventional use in proof theory, in which proof complexity refers the efficiency of proof system (Cok & Reckhow 1979)
5

---

Published as a conference paper at ICLR 2025
from any of the axioms and that the problem has five (declarative) sentences. We distinguish two forms proof-tree shapes: We say that a proof tree is linear if each of its proof steps takes at most one premise that is not axiom, and nonlinear otherwise _ In other words, every step in the solution to linear problem uses only at most one fact not directly present in the problem; see Fig. [ an example. The proof tree given in Fig. is nonlinear because the proof step that deduces the answer uses two premises that are not themselves axioms. Lastly, we define canonical ordering as the ordering of the leaf nodes when visited from left to right. Note that the sentences in the natural language annotation from Fig. 1 follow the canonical ordering of the proof tree. Informally, this is an easy way to order the sentences in the problem, since matches the ordering of the proof steps. However, there exist multiple alternative orderings in general. In particular, given proof tree that only contains commuta- tive inference rules (see footnote 4), all orderings of the leaf nodes are valid. For instance, swapping the ordering of the first two leaf nodes in the tree in Fig. yields a problem with the same proof.
4 EVALUATION FRAMEWORK
Equipped with the formalism explained in [ we propose an evaluation framework that tests LLMs on MWPs that have arbitrarily complex proots, called MathGAP. In $4.II we explain the generation method behind MathGAP. We then outline how this method can be used for OOD evaluation in $42
4.1 GENERATION METHOD
We propose a method for synthetic generation of problems and their CoT solution explanations. The approach consists of three high-level steps: (i) sample proof tree, (ii) map the logical forms at the leaf nodes into a natural language problem, and (iii) map the proof steps into a CoT solution Fig. gives illustration of these steps; note that the agent properties are abbreviated for brevity.
In detail, the procedure is as follows: We first sample a logical form to label the root of the proof tree. We then sample an inference rule whose conclusion matches the root, and use it connect the root node to child nodes representing the premises of the sampled inference rule. We repeat this procedure recursively for each premise the leaf nodes of the current tree until predetermined stopping criterion has been reached, resulting in proof tree. The stopping criterion is user-specified; it could be, e.g., when the tree has reached certain depth or width. The categorical properties for the logical forms (i.e., agent, entity, attribute, and unit) are sampled uniformly at random from some vocabularies.
We then convert each leaf node into sentence via a natural language template in some order, forming the text corresponding to the body of the problem. The default order follows the proof tree's 's canonical ordering, but other orders are possible as well. The sentences are sampled uniformly at random from a predefined set of templates; some example templates are given in Table The question is converted from the logical form the root of the proof tree and is sampled from a set of interrogative sentences.
In the final step, we generate the CoT annotation for the problem. We first obtain a sequential ordering of the CoT proof steps by visiting the nodes in the proof tree with post-order traversal We then convert each proof step into sentence via a natural language template. The templates used for the axioms are identical to those used in the body of the problem text; that is, the proof steps corresponding to axioms simply repeat the relevant part of the problem text. For the other proof steps, we use templates designed to cxplain the computation step that is performed. Fig. [
'In this paper, the width of the proof tree matches the number of declarative sentences (i.e., all sentences except the question) in the corresponding problem since we restrict the logical forms and sentences to be one-to-one. In general, this need not be the case since single sentence could be written as conjunction of several logical forms. One could also construct problems that have irrelevant sentences which either represent logical forms that will not be used in the proof or have no corresponding logical form at all.
"We note that our notion of linear proof's is distinct from linear logic (Giardd 1987 ).
*Our pipeline is similar to Opedal et al.|s (2024), except that we use a top-down procedure to generate full proof trees, rather than sequential one to generate only axioms. This is a more general approach that enables generation of proofs with arbitrary shapes. e.g., nonlinear ones using comp-eq: eq: see App. D Our pipeline is also similar the method proposed by Jin (0023) for generating causal reasoning prohlems.
"Note that the choice of traversing the nodes in post order corresponds to applying each inference rule as soon as all of its premises available. In addition, post-order traversal visits the axioms in canonical ordering. Alternative orders of tree traversal would violate least one of these constraints.
6

---

Published as a conference paper at ICLR 2025
illustrates the mapping between the nodes in the proof tree and the sentences in the CoT annotation. For instance, the third sentence in Fig. [ explains the conclusion cont (Lucy, 27, apple), which is deduced from the two leftmost nodes in the tree: "So Lucy has 17 + 10 = 27 apples".
4.2 MATHGAP: MATHEMATICAL GENER ALIZATION ON ARITHMETIC PROOFS
With our generation method we can generate new synthetic problems where we control for the structure of the proof and its complexity. This mitigates the risk that the problems are contaminated. MathGAP can be used for various forms of systematic studies on arithmetic reasoning; here, we focus on in-context easy-to-hare -to-hard OOD generalization. Specifically, we select a family of problems of interest (e.g., linear problems under comp inference rules) and generate a train-test split of problems in which problems in the test set more complex than those in the training set (e.g.. requiring deeper proofs).
The generation method can be flexibly adapted. One can add new logical forms and inference rules, restrict the choice of existing ones during sampling, create logical forms for irrelevant sentences (Shi et al. 2023). One can also vary the text templates for the logical forms and inference rules, the vocabularies for agents, entities, attributes, and units, and the range of numbers from which the quantities are sampled. In addition, one may increase lexical and syntactic diversity by using an LLM to paraphrase the templated texts, e.g., through procedure like Mishra et al.js (2024). We do not perform such paraphrasing for the data in our experiments since it could introduce bias (Boguraev) el. 2024 al. 2024) and produce problems that are unfaithful to their formal specifications.
5 GENERALIZING TO COMPLEX PROOFS
We apply MathGAP to study how LLMs generalize from proof demonstrations of simple problems in context to solve problems of higher complexity at inference. We test OOD generalization through several sets of experiments: depth generalization for linear problems ($5.I). D. width generalization for linear problems ($5.2), depth generalization for nonlinear problems ($5.3), b, and generalization to
permutations of the axioms (5.4). Our public code repository provides further problem variations.
Experimental setup. For each set of experiments, the general experimental setup is as follows: We generate multiple test sets of different degrees of complexity with 400 problems in each. We then generate model predictions for the problems in these test sets under four different prompts: (i) A zero-shot baseline prompt, which contains only the test problem; (ii) A prompt of primitive examples, in which all problems contain only one proof step of the same inference rule that is used in the test problem; (iii) A prompt of examples within a range of complexities, all of which are simpler than the test problem: (iv) An in-distribution baseline prompt, containing examples that are of the same complexity as the test problem.
Note that prompts (i) through (iii) evaluate in-context OOD generalization, since the test examples from a distribution that is different from that of the in-context examples. For prompts (ii) through (iv) we generate a new set of in-context examples for every test problem. This is done in order to avoid bias in the results towards any particular set of examples. We set the number of examples to 12, except for the experiments on nonlinear problems for which we use 5-see App. A the rationale. Importantly, the in-context problems are provided together with their CoT solution annotations, as generated through the MathGAP pipeline. For prompt (iii), we make sure that there is at least one problem of every complexity in the range. This prompt can be viewed as a form of in-context curriculum learning (Bengio et al. 2009) [1) except that we randomize the ordering of the examples rather than arranging them in increasing order of complexity. This done since in-context orderings can have large effect performance (Lu al. 2022). and do not want our results to overfit to any particular ordering. More details on the data and prompt are given in App. C
Responses generated using greedy decoding and maximum context length of 4,096 tokens. Model predictions are obtained by extracting the last number occurring in the model output. We report answer accuracies and compute 95% confidence intervals using bootstrap sampling (Efron)
1992). We evaluate the following models on all test sets: Mixtral-8x7B (Jiang et al.] 2024a). Llama3
with 8B and 70B parameters (Llama Team 2024 GPT-3.5 Turbo and GPT-4o (OpenAl 2024
Reellted approaches have been explored in previous work Li ei al.
2022
Liu et al.
2024b).
7

---

Published as a conference paper at ICLR 2025
Linear Problems (Comp)
1,00
0.23
0.00
15
Depth
Depth
Depth
Depth
Depth
Zureants
Surget
In-dbic Bustinn
Linear Problems (Transfer)
1.00
0.00
Depth
Deptth
Depth
Degth
Depth
Zarosthe:
jarm bbv)
(panget
In-dtenbutiion
Linear Problems (Part-Whole)
tratruct
Instruct
0.00
22
12
10
wilth
Widthh
Wilthh
wildh
Wilthh
Zuro-inis:
CCD Sor Mina)
OCD Orangal
-i--iiic Bostion
Figure 2: Answer accuracies for generalization to increasing depth and width for linear problems across models and in-context distributions. Depth is increased using inference rules involving comp and transfer. Width is increased using partwhole. See App. E for example problems.
5.1 LINEAR DEPTH GENERALIZATION
Problem sets. We consider linear problems using both comp and t transfer. The five test sets con- sist of problems with depths between 6 10. As such, the examples provided in the range of setting (iii) described above have depths between 5- We give an example of a problem with depth 6 in App. E.
Results. The results are illustrated in the top and middle rows of Fig. As expected, we observe a decreasing trend in performance as depth increases, for all in-context distributions. Curiously, GPT- 3.5-Turbo is the only model for which adding in-contett examples consistently leads to improvements. For the majority of cases, performance tends to be larger when providing a range of examples of varying complexity as compared only primitive examples. The ranged OOD cases often show com- parable performance to that of the in-distribution context, suggesting that these models benefit from demonstrations of problems with varying complexity. Llama3-8B is an outlier in that its solving accu- racy for in-distribution context is (substantially) lower as compared to other prompts. When inspecting the outputs we see that the model often fails to answer the inference question, recognizing only that has been provided sequence of in-context examples. This could potentially be consequence of how Llama3 is trained, using smaller context windows for the initial stages pretraining (Llama Team 2024 This appears to be remedied by scale since Llama3 70B shows near perfect performance.
5.2
LINEAR WIDTH GENERALIZATION
Problem sets. We test generalization to larger proof width using partwhole problems with fixed depth of The five test sets have widths between 7-11 and the problems in setting (iii) described above have widths between 2-6. Note that this matches the width for the problems in $5.1 for those problems the width was equal to the depth plus one. App. H.2 gives an example problem.
The results are shown in the bottom row of Fig. 2 Apart from the GPT models, the perfor- mance is generally lower as compared to deep problems with the same number of sentences ($5.I). often with sharper decreasing trend. Such is the case even for Llama3 70B, which performed almost perfectly in the linear depth setting. This suggests that it is more difficult to generalize terms of width than in terms of depth, which is somewhat surprising considering that these problems only have
Results.
8

---

Published as a conference paper at ICLR 2025
Nonlinear Problams
Diveni 200 girnattdd
000
Figure 3: Answer accuracies for generalization to increasing depth and width for nonlinear problems across models and in-context distributions. Depth is increased using inference rules involving comp and comp-eq. The in-distribution contexts were too large to fit in context for the Llama3 models and GPT-3.5 Turbo at some of the higher depths. App. F shows results on ol -preview and DeepSeek-R1
one proof step. For GPT-3.5 and Mixtral- Mixtral-8x7B, we observe that providing in-context examples im- proves accuracy over zero-shot in most cases. As for linear depth generalization, it seems that having a range of OOD examples of varying complexity is superior to only including primitive examples. For the Llama models, adding OOD in-context examples often leads to lower performance over zero-shot.
5.3 NONLINEAR DEPTH GENERALIZATION
Problem sets. We generate test datasets containing nonlinear problems with comp and comp-eq which have a similar form as the problem in Fig. The procedure is explained in App. Dop..Dam and more complex, depth 3 example is given in App. Thus, the ranged OOD context setting contains examples of depths between 1-2.
E.3 We generate test set for each depth between 3-6.
The results are shown in Fig. 3 All models exhibit a performance trend that tends to zero depth increases, albeit at different rates. For Mixtral-8x7B, Llama3-8B, and GPT-3.5 Turbo, the performance decreases rapidly across all context modes. These models benefit from in-distribution contexts for low depths, and in some cases, from a range of OOD examples. LJama3-70B and GPT-4o are more robust to this type of OOD distribution shift 11 but for the deepest problems their performance tends to zero as well. Thus, the models are able to generalize to some extent but the error rate increases with complexity, suggesting that LLMs are noisy reasoners (Prabhakar et al. 2024). Moreover, it worth noting that, for many cases, providing in-context examples does not significantly improve performance over zero-shot. On the other hand, for lower depths, it is usually beneficial to provide in- distribution contexts. We perform an additional analysis on ol -preview and DeepSeek-RR RI in App. F
Results.
Comparison to linear problems. Nonlinear problems are more difficult than the linear ones, even
when controlling for the number of axioms given in the problem (i.e., its width). To see this, note that a nonlinear problem with depth 3 has 10 axioms (see App. D). This is the same number of axioms as the linear comparison and transfer problems S.11 with depth 9 and the linear wart- problems ($5.2) with width 10. Comparing the results across those problem sets, we observe considerably lower accuracies for the nonlinear problems in most cases. We believe there are two main explanations for this: (i) Solving nonlinear problems requires keeping intermediate conclusions in memory while proving other intermediate conclusions, before being able to them for further proof steps. It is thus less predictable where the relevant tokens will be located within the context window as compared to linear problems, in which intermediate conclusions are used immediately Nonlinear problems use the comp-er rule, which is most likely less frequent in the training set. In- specting the model outputs we find support for both of these explanations errors show up both when a previously deduced conclusion is needed as well as when a comp-es -amp-eqie is required in proof step.
new proof step. (ii)
5.4 ORDER GENERALIZATION
Problem sets. LLMss are known to be sensitive to changes in the order of axioms in logical and math-
ematical reasoning problems (Chen et al. 2024 Eisape et al. 2024p. Here, we present a systematic analysis of their sensitivity particular form of order permutation. We consider linear, left-lcaning
" Note that GPT-4o performs better on depth 6 problems than depth 5 problems under the in in-distribution prompt. Although we provided the same model id as in the other cases-which according to the documentation points the original cgpt-4o-2024-0--1 snapshot- this particular test was performed later point in time.

---

Published as a conference paper at ICLR 2025
Permuted Linear Problems
1.0
0.23
0.00
Distance
Distance
Distance
Distance
Distance
02D tranged)
Bullon
Figure Answer accuracies for generalization permutations across models and in-context distributions. We measure complexity as the distance of the movement of a sentence to the beginning of the problem, as compared its position the canonical ordering of the problem.
proof trees using comp that have depth 5, and generate problems that deviate from the canonical, left- to-right order of the leaf nodes. In particular, we pick one of the leaf nodes to visit first, and then visit the remaining leaf nodes in left-to-right order. This has the effect moving one of the sentences the canonically-ordered problem text to the beginning of the problem. An illustration is shown in Fig. and App. [E.4 gives further examples. We characterize complexity as the distance the movement, with the hypothesis that a greater movement from canonical ordering constitutes more difficult problem. We create five test sets with movement distances between 1-5. We do not include a primi- tive OOD in-context distribution since the notion of primitive problem is ill-defined in this case. The range prompt includes movements of all distances between 1-5 5 except the one present in the test set.
Results. The results are shown in Fig. 4 First, note that all models show lower performance perturbed problems as compared to non-perturbed ones (compare to Fig. 2 suggesting that they sensitive to simple perturbations of axiom orderings. Second, there nonlinear relationship between accuracy and distance of the movement, in contrast to the monotonically decreasing relation we expected. More specifically, the performance is the highest for the problems where the sentence is moved from near the beginning or the end of the problem examples gives a larger boost performance for short and long movements, as compared medium- length ones. Inspecting the model outputs, we observe that the models often make logical mistakes when the sentence that has been moved is needed for the subsequent proof steps. These kinds of mistakes appear to be more common than arithmetic errors.
In addition, including in-context
6 CONCLUSION
This paper introduced MathGAP, a framework for evaluating LLMs on math word problems with proofs that arbitrarily complex. MathGAP can flexibly generate synthetic arithmetic word problems with controllable proof tree characteristics. Importantly, this enables studies of easy-to- hard OOD generalization that avoid issues of data contamination, since a train-test split where the distributions are different can be created programmatically. In particular, the test set can be made arbitrarily more complex than the training set. We applied MathGAP to study whether LLMs can learn from simple examples in context to solve more complex problems in test set. All models showed decrease in performance as complexity of proof trees increases through depth and/or width. The L.Mss struggled more with wider problems than with deeper problems, and were worse at solving problems with nonlinear shapes as compared to linear ones. We also found that LLMs are sensitive to the order in which sentences are presented. We further demonstrated that we can use MathGAP to construct problems that are challenging even for state-of-the tfr---rrtttt reasoning models like OpenAl ol and DeepSeck-RR R1 Re-see
see App.
Code to generate problems with MathGAP, including problem types beyond those considered in this paper, can be found in our public code repository. Finally, see App. G for discussion limitations of the present work.
12This is consistent with findings retrieval- -uggeeneed generation, for which models are better at using information that occurs near the beginning end of the prompt (Liu al. 2024a).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Alessandro Stolfo, Moritz Miller, Ryan Cotterell, Shehzaad Dhuliawala, Yanick Zengaftinen. and Yilmazcan Ozyurt for useful discussion. Andreas Opedal acknowledges funding from the Max Planck ETH Center for Learning Systems.
REFERENCES
Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behba- hani, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning. In A. Glober- son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Ad- vances in Neural Information Processing Systems, volume 37, pp. 76930-76966. Curran Asso- ciates, Inc.. 2024. URL http::::: htt.://ppw/prreeiii........oaoo. wwtp:ur....pccccpppppaeeaeaaeapa fiwepepppeeeeeeppeeeeeeeaaee pdf
Shengnan An, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Jian-Guang Lou, and Dongmei Zhang. How do in-context examples affect compositional generalization? In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 11027-11052, Toronto, Canada, July 2023. Association for Computational Linguistics. BRRLhtpps: http:ss:/aaaaatthhhcco http///ccaanntooo...oooo 2023. acl- long 618
Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ra- masesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Ex- ploring length generalization in large language models. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.). Advances in Neural In- formation Processing Systems, volume 35, pp. 38546-38556. Curran Associates, Inc.. 2022. URL https::/proceeiiggs. www.eediiinnnnu...nnueee nww. ctppppperppcppappe__rrreeeeee http://wweeacooo...ooooooooooeaa
file/
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning. ICML '09,
pp. 41-48, New York, NY, USA, 2009. Association for Computing Machinery. URL https:
Sasha Boguraev, Ben Lipkin, Leonie Weissweiler, and Kyle Mahowald. Models can and should em- brace the communicative nature of human-generen math. In The 4th Workshop Mathematical Reasoning and Al at NeurlPS 24, 2024. URL https: /apps://arxiiii //txavaxvvv///////////aa///aaav. org/ aba/ 240.. 17005
Nasim Borazjanizadeh and Steven Piantadosi. Reliable reasoning beyond natural language, 2024.
URL https:://axxi...ttp org. aa////b//2224 11373
Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilé Lukošiūté, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran- Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Liane Lovitt, Nelson Elhage. Nicholas Schiefer, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Robin Larson, Sam McCandlish, Sandipan Kundu, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Timothy Tellcen-Lawton. Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds Ben Mann, and Jared Kaplan. Measuring progress on scalable oversight for large language models, 2022. URL URLhttts: https: ftctts://arxiv.coo http://wwwww.../////////222 03540
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. Weak-to- strong generalization: Eliciting strong capabilities with weak supervision, 2023. URL. https
7/arxiv. www.eorg/aae awwg///b/// 23322.2 09390
Xinyun Chen, Ryan Andrew Chi, Xuezhi Wang, and Denny Zhou. Premise order matters in reasoning with large language models. In Forty International Conference on Machine Learning, 2024.
URL https:: / rxxiv xiv. org/
08939

---

Published as a conference paper at ICLR 2025
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchrr Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL
abs/ 2t1./210...0 14168
Stephen A. Cook and Robert A. Reckhow. The relative efficiency of propositional proof sys- tems. Journal of Symbolic Logic, 44()3,6--00 1979. URL http:ttp:/ /wwp///www/// cs wttp://ss...oootttttt.oooo wtw.ooto..o...........o... ca/
DeepSeek-AL, Deepseck-RI; mecentivizinng capability in LLMs via reinforcement learning. 2025.
Chunyuan Deng, Yilun Zhao, Xiangru Tang. Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume Long Pupers), pp. 8706-8719, Mexico City, Mexico, June 2024. Association for Computational Linguisties. URL
www.:aannnnn.oooorrr//2/222// naacl -longnng.....................l 482.
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1286-1305, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL htt.t:://///ee antho logy org/2021. emn main.
Moens, Xuanjing Huang, Lucia Specia,
Nouha Dziri, Ximing Lu, Melanic Sclar, Xiang (Lorraine) Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena Hwang, Soumya Sanyal, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality. In Oh, Naumann, A. Globerson, Saenko, M. Hardt, and Levine (eds.), Advances in Neural Information Processing Systems, volume 36. pp. 70293-70332 Curran Asso- httppppppperrlleeppppppppppppp
ciates, Inc., 2023. URL httphtttps:/prroceednngg
2023 f (2aag//ee22222122233322255222222
Bradley Efron. Bootstrap methods: Another look at the jackknife. In Breukthroughs in Statisticy:
Methodology and Distribution, pp. 569 Springer, 1992.
com/ whtptet:/1..110////.
1110--11-611---10 4380-9 41
Tiwalayo Eisape, Michael Henry Tessler, Ishita Dasgupta, Fei Sha, Sjoerd van Steenkiste, and Tal Linzen. A systematic comparison of syllogistic reasoning in humans and language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 1: Long Papers), pp. 8425-8444, Mexico City, Mexico, June 2024. Association for Computational Linguistics. URL http:pps//aaallatoooooooooo hwwp:hhhhwoolloo...o2eee
nww. wttp:///2222....1111o1....
466
Yanai Elazar, Akshita Bhagia, Ian Helgi Magmusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hannaneh Hajishirzi, Noah A. Smith, and Jesse Dodge. What's in my big data? In The Twelfth International Conference on Learning Representutions, 2024. URL https::t//// : / //axxvivvv org wttp://orrra/////////// abs//311/.0.0 20707
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, 2023. /ttptpps:///xii....m/
Gerhard Gentzen. Untersuchungen über das logische schlieBen. I. Mathematische Zeitschrift, 39(1): 776-210, Dec
Jean- Yves Girard. Linear logic. Theoretical TheooeticclCCppppeeeSSriitiitiee Sciencee Scietreee 1987. URL https 7/ w/www. scceeeediieiiiic rect. com/ httpo//..ooo..cccaccooooooaaoe/a
12

---

Published as a conference paper at ICLR 2025
Peter Hase, Mohit Bansal, Peter Clark, and Sarah Wiegreffe. The unreasonable effectiveness of easy training data for hard tasks. In Lun-Wat Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 7002-7024, Bangkok, Thailand, August 2024. Association for Computational Linguisties. URL
ogy. org/ / 2024 aac...ccl.. long. 3778
Yi Hu, Xiaojuan Tang, Haotong Yang, and Muhan Zhang. Case-based or rule-based: How do transformers do the math? In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning. volume 235 of Proceedings of Machine Learning Research, pp. 19438 19474. PMLR, 21-27 Jul 2024. URL http:tttss::/rrceeeeiiie waaaagssll...... weress/v233
hu2 4n. html
Yinya Huang, Xiaohan Lin, Zhengying Liu, Qingxing Cao, Huajian Xin, Haiming Wang, Zhenguo Li, Linqi Song, and Xiaodan Liang. Mustard: Mastering uniform synthesis of theorem and proof data. In The Twelfth International Conference on Learning Representations, 2024. URL aittps://aae //axii/// org/ abs/ 2402. 08957
Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5075 5084,, Singapore, December 2023. Association for Computational Linguistics. URLhttps:
/LLLttpps//claatthooooogs /ataaaaannttttooooeeee org/ 2023. emnlp-main
308
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril. Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024a. httpstt:: wwtp/awwww.....aa.aa.aooo/o. abs 220./4201.00 04088
Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang. Tanwi Mallick, Weijie Su, Camillo Jose Taylor, and Dan Roth. A peek into token bias: Large language models are not yet genuine reasoners. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 4722 4756, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. wtg./lloyooooooooooooooooooooooe 2024 emnlp-main.272
URL
Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng LYU, Kevin Blin, Fernando Gonzalez Adauto, Max Kleiman-Weiner, Mrinmaya Sachan, and Bernhard Schölkopf. Cladder: Assessing causal reasoning in language models. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 31038-31065. Curran Associates, Inc., 2023. URL https: beetts::/proceeddinei www.iiin......u..uuuu.oo wew.esrriip www.eeewwrsssppppeeeeeeeeeeeeeee wc/paper locerr_fiiee files//ape::20233 file/ 631 2et.://1.....3.111.000000
Pww.eaaaaaaaereecceeeeeeaa pddf
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratík Ringshia Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp. Robin Jia. Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in NLP. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4110-4124, Online, June 2021. Association for Computational Linguistics. URL https://
aclant aaanhhoooogy.... orgg / 0221 naacl ma i nn324
Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Ana Brassard, Masashi Yoshikawa, Keisuke Sak- aguchi, and Kentaro Inui. Do deep neural networks capture compositionality in arithmetic reasoning? In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp-
13

---

Published as a conference paper at ICLR 2025
1351-1362, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. URL https://acl http::///ccaaattooooooorrrr///// wtt.:////..................... eacl-main. 98
Changqun Li, Linlin Wang, Xin Lin, Gerard de Melo, and Liang He. Curriculum prompt learning with self-training for abstractive dialogue summarization. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 1096- 1106, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
emnlp-mat
72
Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shorteuts to automata. In The Eleventh International Conference Learning Representations, 2023. URL http://////aa.a...ooooaaa///////
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transac- tions of the Association for Computational Linguistics, 12:157-173, 2024a. URI. https
//aclanthology wtwp:nthhwoooyooog/222222220 tacl-1.3
Yinpeng Liu, Jiawei Liu, Xiang Shi, Qikai Cheng, Yong Huang, and Wei Lu. Let's lear step by step: Enhancing in-context learning ability with curriculum learning, 2024b. URL https
/ //arxiv. org/ abs/ /2402.2422222 10738
Llama Team. The llama 3 herd of models, 2024. URL 21783
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086-8098, Dublin, Ireland, May 2022. Association for Computational Linguistics. URL org/ 2022 acl www.allll................ 556
htttps://aclanthott
Anna Mészáros, Szilvia Ujváry, Wieland Brendel, Patrik Reizinger, and Ferenc Huszár. Rule ex- trapolation in language modeling: A study of compositional generalization on ood prompts. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet. J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 34870-34899, Curran Asso- neur iiwprrpppp.. ps cc wwwpppprpppppppeeeeeeeeeeeeeaeee files/paperr
ciates, Inc., 2024. htttttttt///:/roeeeeeet 2024/file/30/ 612886680300388888232 9ttp:ee222411fffff----p-e--e wow.apprr-oooofencceeoo
Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. In The Thirteenth International Conference on Learning Representations,
2025. URL arwiv. org/ abs 02210.40.2222
Shubhra Mishra, Gabriel Poesia, Belinda Mo, and Noah D. Goodman. Mathcamps: Fine-grained synthesis of mathematical problems from human curricula, 2024. URL httl:ttss///axxv.oooo
abs/2407 2407 00900
Pearla Nesher, James G. Greeno, and Mary S. Riley. The development of semantic categories for addition and subtraction. Educational Studies in Mathematics, 13:373-394 1982. URL
https://api.
Andreas Opedal, Niklas Stoehr, Abulhair Saparov, and Mrinmaya Sachan. World models for math story problems. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 9088-9115, Toronto, Canada, July 2023, Association for Computational Linguistics. URL https: /aac:s://ccaaaaa http///aaaannntoy..ooo 2023. www.inggnnnl.l.....
Andreas Opedal, Alessandro Stolfo, Haruki Shirakami, Ying Jiao, Ryan Cotterell, Bernhard Schölkopf, Abulhair Saparov, and Mrinmaya Sachan. Do language models exhibit the same cognitive biases in problem solving human learners? In Proceedings of the 41st International Conference on Machine Learning, ICML'24, 2024. URL RoLhhttss:. ///ariv....iv oxq/ etg/abs////eeooooooee..//e//aaar abs 2401 18070
14

---

Published as a conference paper at ICLR 2025
OpenAl. Gpt-4 technical report, 2024. UBL.Itpps///rxxv/.r/////////
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tun Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, pp. 2080 2094, Online, June 2021. Association for LinguttiionalLnnguiiiissss.. Linguistics. URL/tttps://aclle
otg. 202/ www.:22mmmaammaaaaaaaaa
168
Frank Prenning. Natural deduction. Lecture notes 15-815 Automated Theorem Proving, 2004, URL https: / www. Cs. cmu eet.eeddu///. wtcduuufp//oouseeeeeeeee attp:haaddats//chtaaaaaaaeee
Akshara Prabhakar, Thomas L. Griffiths, and R. Thomas McCoy. Deciphering the factors influ- encing the efficacy of chain-of-thought: Probability, memorization, and noisy reasoning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 3710-3724, Miami, Florida, USA, November 2024.
Association for Computational Linguistics. URL hitps: httpss//aalatttlllogg//.ooo/o/// http://aaaaaaatoloooooooo///////
Fingings-emn. 212/
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot numerical reasoning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.). Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 840-854, Abu Dhabi, United Arab Emirates, December 2022, Association for Computational Linguisties. URL https:: https////aaaatthoooooooo
/tlegg/22220 2022. wtt.022...inniie wiwniinnnnnnnnn. 59
Mary Riley, James Greeno, and Joan Heller. Development of Children's Problem-Solving Ability in Arithmetic, pp. 153-196. Learning Research and Development Center, University of Pittsburgh, 01 1983. URL
gov wwt.l/ooo//dddd2221111101
Oscar Sainz, Jon Campos, Iker Garcfa-Ferreny Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNIPP 2023, pp. 10776-10787, Singapore, December 2023. Association for Computational Linguistics. URL htttp::/aclaathhant wtwpooooooo.o.gg/22/2/22/
Findings-ennip 722
Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of thain-of-thought. In The Eleventh International Conference on Learning Representations, 2023.
URL httttt:/pp//atiti org/ abs 01240
Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Mehran Kazemi, Najoung Kim, and He He. Testing the general deductive reasoning ca- pacity of large language models using ood examples. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural In- formation Processing Systems, volume 36, pp. 3083-3105, Curran Associates, Inc., files//apprr22233// pdf
2023. URL https://prooceddiin www.:ee.eerreeepewe....eeo [LLe0999299113360033339401122222
Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn algorithm? generalizing from easy to hard problems with recurrent networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 6695- 6706. Curran Associates, Inc., wttp:///pp../erreeeei......rooo.
pdf Paulo Shakarian, Abhinav Koyyalamudi, Noel Ngu, and Lakshmivihari Mareedu. An independent evaluation of ChatGPT on mathematical word problems (MWP). In Andreas Martin, Hans- Georg Fill, Aurona Gerber, Knut Hinkelmann, Doug Lenat, Reinhard Stolle, and Frank van Harmelen (eds.), Proceedings of the AAAI 2023 Spring Symposium Challenges Requiring the Combination of Machine Learning and Knowledge Engineering (AAAI-MAKE 2023). Hyatt Regency, San Francisco Airport, California, USA, March 27-29, 2023, volume 3433 of CEUR
15

---

Published as a conference paper at ICLR 2025
Workshop Proceedings. CEUR-WS.one 2023. httL:ttp:////uusss.ooote
paperr.pdl
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31210-31227. PMLR, 23-29 Jul 2023. URL https: https::/pproceddingg: ml2 http://e....n..........eeeeeo/ea wttppppsssss...pppeepe
html
Elsbeth Stern. What makes certain arithmetic word problems involving the comparison of sets so difficult for children? Journal of Educational Psychology. 85:7-23, 1993. URL https
//psycnet. wwwpppp..re...or../eeee/eeeeeeee 01193--2116-000
Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf. and Mrinmaya Sachan. A causal framework quantify the robustness of mathematical reasoning with language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 545- 561, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https //aclanthology org/ 2023 long.
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang
Gan. Easy-to-hard generalization: Scalable alignment beyond human supervision. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https
// /arxiv.
09472
Jonathan Thomm, Giacomo Camposampiero, Aleksandar Terzic, Michael Hersche, Bern- hard Schölkopf. and Abbas Rahimi. Limits of transformer language models on learning to compose algorithms. In A. Globerson, L. Mackey, D. Belgrave. A. Fan, U. Paquet, Tomezak, and C. Zhang (eds.), Advances Neural Information Pro- cessing Systems, volume 37, pp. 7631-7674. Curran Associates, Inc., 2024. URL
cc/ wttphlppp.ccppppppeeeeeeee http://peer..lleeepppp////////// 0e 0a299711113300044222220000111112
https://proceedinggs papp Confereneo
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning large language models. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). Advances in Neural Information Processing Systems, 2022. URL, httpt:: httttps://axxi..orggggoo awwgg//as////// 2201 1903
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physies of language models: Part 2.1, grade-school math and the hidden reasoning process. In The Thirteenth International Conference on Learning Representations, 2025. URL htth:t/::://
org /ta.////43 20311
Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference Learning Representations, 2024. URL http:::///saatta www.vv.rrgaaaaa. /abbb.//299.....
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. A careful examination large language model performance on grade school arithmetic. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 46819-46836. Curran Associates, Inc., 2024. URL /ttp neurippp cc/ cttpcc//pppp// cc/paper_fcap files/ Paper Datasets.
and Benchmarks beenchaarks_Track.. Track.pdl
Yi Zhang, Arturs Backurs, Sébastien Bubeck. Ronen Eldan, Suriya Gunasekar, and Tal Wagner. Unveiling transformers with LEGO: synthetic reasoning task, 2023. htLLtt/t::///// ://axiii///xxv
2206. 04301
16

---

Published as a conference paper at ICLR 2025
Problem Text
World Model
Isabella has 17 apples.
containerllsabell. 17, apple:;
Eucy has 10 more apples than Isabella.
comparisonil.ucy Isabella, 10, applet;
John has apples.
containeriloho, 11, applet;
Emily has 19 apples.
containerlEmmlly 19, applei;
The number apples that Lacy has more than Sam the same as the difference between the number of apples that John has compured Emily.
comp-ey/Lacy, Sam, Emily, John, applet:
containeriSam, 9, applet;
contuinerSSam, 19, applet;
How apples does Sam have?
Answer: 19
Figure 5: World model (right) of math word problem (left). Each sentence in the problem text is rep- resented by a logical form which consists of a predicate with property arguments. The logical forms in the body are used axioms the proof of the problem, as shown in Fig. The sentences and logical forms labeled (1) through (5) are the body of the problem, (6) is the question and (7) is the answer.
Mingyu Zong and Bhaskar Krishnamachari. Solving math word problems concerning systems of equations with GPT-3. Proceedings of the AAAI Conference on Artificial Intelligence, 37(13); 15972-15979, Sep. 2023. URLtttss:://73 aaai www.aai.oggiinnneo view/26896
A INTUITION ON LOGICAL FORMS
A cont logical form expresses the fact that an agent owns particular entity in some quantity (e.g.. "Alice has 5 apples") "). A comp logical form expresses how much more some agent owns of a particular entity than some other agent (e.g., "Bob has 3 fewer apples than Alice"), A transfez logical form expresses the transfer a certain quantity of an entity from one agent to another (e.g., "Alice gave Bob 3 apples "). A partwhole e logical form expresses superset relation between the combined entities of several agents (e.g., Alice and Bob combine their fruits). Finally, a comp-ea logical form expresses that the quantity of particular cemp between two agents equal the difference of the quantities of the entity of two other agents (e.g.. "The number of apples that Alice has more than Bob the same as the difference berween the number of apples that Charlie and David have").
Fig. 5 gives an example problem text along with its world model.
B DEFINING INFERENCE RULES WITH NATURAL DEDUCTION
Our inference rules can be written using the primitive inference rules present in natural deduction (Gentzen 1935 Pfenning 2004) as applied first-order logic expressions; see Opedal al. (0023 App. D) for details on how the formalism described in is form of first-order logic. In this section we illustrate example using the inference rule
cont (a, 31 e) comp (b, a, q2r e)
cont (b, qi q2 e)
involving comp. This inference rule is composed of rules in natural deduction as follows:
1. 3r (Owne r (w1, a) (Measure q1 5 Vv E (n)e(z)) by Axiom (1*t premise, definition of container in Opedal al. 2023 App. D.2.1).
2. Owner(M,, Measure(M. 9n) Var E v).e(z) by I-elimination
3. Jeg (CompariisonAdd(e)) A Source(e,, 01) A Target 02) A Time(v1) = Time(cz) A 3r(Arg(e).)) Measure(r, q2) Vz E n.e(z))) by Axiom n2 premise, definition of comparison as Opeoal et al. 2023 App. D.2.2).
4. comprrisonddell) ar(Arg(e1,r) A Measure(r, 92) Va E re(()) by Feelimination
vn Target(e), (2) A Time(vi) = Time(us) A
17

---

Published as a conference paper at ICLR 2025
5. 3r(Arg(e1, Mhaauueerrss ga) Vr E ene(z)) by A-elimination
6. Arg(er, A Measure(fs Yr E r.e(z) by 3-elimination
[
7. Arg(e1:7) A Va E (.e(z) by A-elimination
8. Conparisonndainn Arg(e1. A E (e(()) by A-introduction Cetarggeeeeeeee
9. Vetr(Coomarisooadddg tetriCoomaarssnnaael A Arg(e,r) A Ve E r.e(z) + (Taoae(T v) A Owner(v,o) A Measure q) A Vr v.e(z)) by Axiom (theorem on the existence of containers from existing relations as in Opedal et al. 2023 App. D.4).
( Owner(v,o) A A Maeeesaaeaaaeeeeeeee
10. ComparisonAdd(es) comparisonAddeel) A Arg(e1,7) A Vo ne(x) + (Targe(ea A Ar E v.e(x)) by V-eliminatioo (9
11. Briool((Trret((e) (oneertwwoeggeeeeee..oooooooooe
-) MaaaarrrrrooeoMauueeeeeeeee a A)v v.e(r)) by -eliminatio
10
12. Target U2) (wner(uy, b) Measure(r2, q.) A Vo E 12.e(z) by F-elimination
13. Owner(e2, b) Measure( q) Vo E (2))(2) by A-elimination A Arg(e,r) A Source v.). A
12
14. Ve, www.Hyy,yymmmw/coo.e wtwp:/mmwwccmpprrssoonddeo Target(e, A Measure(v,, m.) A Measure(U; my (
m.
me) by
Axiom (theorem defining the semantics of comparison as in Opedal et el. 2023
App. D.3).
15. Comparisol. ComparisonAdd(e) Measure(U, 41) A Measure("2, q:) + + 92 = q by Y-elimination 14
Arg(ei,))
Source(e1, u1) A Ta reret((1, 02)
16. Measure ()) by A-elimination
2
41 17. Measure ) by A-elimination
13
18. ComparisonAdd((e) (omparisonAdd((e)) A Arg((1, by A-climination
8
4)) Aargget v2) by A-elimination
4
20. ComparisonAdd((e) A Arg(ei,r) A Source((1 U1 A Target 02) A
M) A (Va Meaauurr qt) by A-introduction [4 16
17 18
19
21. 91 + 92 = by -llmminatio
15 20
22. Owner(e2, b) Measure 41 + 42) A Va v2.e(x) by eliminattio
13 21
23. (uy(Own b) A Measure(ez, q1 + 42) A Va E v2.e(z)) by 3-introduction 22
Note that the conclusion (step 22 the first-order logic equivalent of cont (b, q1 + q2, e), which is the conclusion of the comparison inference rule.
C MORE DETAILS ON EXPERIMENTS
Generated problems. The quantities are sampled uniformly at random from the range 2-20. We use handwritten list of English-language names for most problems, with the exception of the larger nonlinear problems for which we use a larger list of 4,945 predominantly English- names from https://giuub..oom/ooooo o name For the entities we use handwritten word https//////giieeeee
list with 51 items. We restrict each problem to have entity, We determine uniformly random whether the problem has attributes, units, or neither of them.
Prompts. We use 12 in-context examples for all test sets except for the nonlinear problems, for which we use 5. We use fewer in-context examples for nonlinear problems since the number of sentences in a nonlinear problem increases exponentially with depth. The number 12 was deemed large enough so that additional examples would have a negligible positive impact on performance; see, e.g.. Agarwal et al. (2044 Fig. 7). Moreover, in preliminary experiments we found that the number of in-context examples had little effect on performance. The in-context examples are written using the pattern "Q: {probl 1 em) AnA: (CoT solution]/m
expontaee
18

---

Published as a conference paper at ICLR 2025
Models. The maximum context windows for tixttral-x7B.. Llama3 8B. Llama3 70B, GPT-3.5 Turbo and GPT-4o are 65k, 8k, 8k, 16k, 128k, respectively. Combined with maximum number of output tokens of 4,096, the models have theoretically sufficient context size to solve all test problems. The largest test problems (nonlinear depth 6) have around 2.5k tokens, including CoT annotation.
For GPT-3.5 Turbo and GPT-4o we used "gpt-3.5-turbo-011 and (opt--o-2022-05-11 as the model ids for all experiments.
D NONLINEAR COMPARISON PROBLEMS
In this section we explain why we have introduced the comp eq predicate to generate nonlinear trees. We also give the procedure for generating the nonlinear problems used for the experiments in $5.3
Motivating example. We first motivate why introduce the comp-eq predicate to build proof trees with nonlinear shape of arbitrary depth. Consider the following problem which is based solely on logical forms with cont and comp: "Alice has apples. Bob has 3 more apples than Alice. Charlie has 10 apples. David has 2 more apples than Charlie. How many more apples does David have than Bob? ? This problem represented by the following nonlinear proof tree of depth 2:
cont (A, 5, a) comp (B, A, 3, a) cont (C, 10, a) comp (D, C, 2, a) cont (B, 8, a) comp (D, B, 4, a)
cont (D, 12, a)
Now, say we are following the top-down generation procedure described in $4.1 and wish to expand the tree to higher depths. However, the two leaf nodes labeled with comp are impossible to expand further under the given inference rules. For instance, the comp (B, A, 3, a) would deduced by the two premises cont (A, 5, a) and cont (B, 8, a), but those two logical forms already present elsewhere in the tree and are thus not available. This leaves only the two cont nodes for further expansion. However, any further subtrees of the cont nodes will be linear, since none of the comp nodes generated along the way will be able to be expanded [3]
The inference rule with comp-eq addresses this issue. To this, replace the subtree rooted at cont (B, 8, a) in the proof tree above with some subtree that uses an inference rule with comp-es e.g..
cont (A, 5, a) comp (E, F, 4, a) comp-eq eq (B, A, E, F) cont (B, 8, a)
Note that, in this case, both cont (A, 5, a) and comp (E, F, 4, a) can be expanded. In particular, comp (E, F, 4, a) introduces two new agents E and F, so it can be expanded similarly to the root node from our example above.
Procedure. We want to sample nonlinear proof tree with depth D. We start by sampling either a cont node comp node uniformly at random to determine the root of the tree. Next, we sample an inference rule for which the root logical form is the conclusion. We then sample inference rules from the child nodes in breadth-firse order until all leaves except the ones labeled with comp-ec (which cannot be expanded) have depth D. For comp nodes, there is only one inference rule possible (the third row in Table 2). For cont nodes we sample comp rule (i.c., the first row in Table if the cont node has depth D 1, and a comp-eq rule (i.e., the fifth row in Table 2 otherwise. This ensures that we can continue to grow the tree nonlinearly until the specified depth, following the reasoning given in the paragraph above.
Note that the width of the proof trees generated with this procedure will be the number of leaf nodes in a perfect binary tree with depth D, i.e., 2D, plus the number of nodes labeled with comp eq. Thus, a depth 2 problem will have a width of either 4 or 5, a depth 3 problem will have a width of 10.
This reasoning assumes that only the inference rules with cont. and comp from Table Deeper nonlinear problems could he achieved by using the partwhol e rule well.
19

---

Published as a conference paper at ICLR 2025
Proof tree
cont (2, (2,27,ayple)
coup (L, 10,, aapple)
cont cont(L, apppl))
comp (L,3,11,appp)
cont (J, appple)
comp 9J, 9,pple))
Tapple)
Word problem
Chain-of-Thought Reasoning Trace
Isabelli has apples. Lucy has 10 more apples than Iaabella, Lacy has 11 more applex cham John. Emiily has fewer apples than John.
Isshells has apples. Lucy has more apples chan lashells. @ So Lucy has 17 10 27 apples. Lucy has more apples than John. @0 John has 27 it apples. Emily has 9 lewer apples than John. So Emily has 16 9 apples.
How many apples does Emiily have?
Figure 6: Example of linear problem using comp with depth 3 and width 4. This problem is linear because each parent node in the tree has at most one child that is not leaf node. Compare to the nonlinear problem shown in Fig. [
and depth 4 problem will have width of either 20 or 21. In general. the width W W(D) for proof tree depth D is:
D-2
W(D) = 2 + Σ feld),
d=0
where fo(d) gives the number of comp eq nodes function of the level d, following the recursion
{1 root is cont
fe(0) = 0 root is comp
1 if root is cont
fe(1) 2 root is comp
f((d)= = fold - 1) + 2fc(d 2), 2 2.
E EXAMPLE PROBLEMS AND ILLUSTRATIONS
In this section we provide example problems for each set of experiments ($$ 5.1 to 5.4
E.I EXAMPLE LINEAR COMPARISON PROBLEM (DEPTH 6)
Jackson has 16 red bottles of soap. Jackson has 10 more red bottles of soap than Abigail. Joseph has 18 more red bottles of soap than Abigail. Joseph has 14 fewer red bottles of soap than James. Michael has more red bottles of soap than James. Ryan has 16 fewer red bottles of soap than Michael. Mia has more red bottles of soap than Ryan. What is the number of red bottles of soap that Mia has?
E.2 EXAMPLE PARTWIIOLE WIDTH PROBLEM (WIDTI 7)
Emily has apples. Lily has bananas. Abigail has 9 bananas. Benjamin has 11 grapes. Christopher has 20 apples. Mila has 16 grapes. Sophia has 11 watermelons. If everyone sums up the fruits that they have, how many fruits does everybody have total?
E.3 EXAMPLE NONLINEAR COMPARISON PROBLEM (DEPTH 3)
Ella has yellow plates. Ella has 19 fewer yellow plates than Jacob. Evelyn has 16 yellow plates. Daniel has 10 yellow plates. The number of yellow plates that Emma has more than Jacob is the same as the difference between the number of yellow plates that Evelyn has compared to Daniel. Lucy has 2 yellow plates. Amelia has 6 more yellow plates than Lucy. Layla has 17 yellow plates.
20

---

Published as a conference paper at ICLR 2025
Proof tree
cont (1,17, apppppl) comp (L, appple)
27, applle)
comp (1,J,11,
cont(J,16,,ap))))
9,,
cont (2,7 7.pple)
Word problem (movement distance: 2)
Chain-of-Though Reasoning Trace
1 Lacy hax 11 more apples than John. @ hobhlll hax apples, @ Lucy has more apples than Isabella, @ Emily has lever applex than John.
lauy more apples than John. @ lesbells spples. @ laaay se 10 more apples tian foobulii,, So Lucy has 17 10-22 spplev. did John has 27- = apples. @ Emmily has fewer apples than John. So Emily has Is 9- 9- apples.
How many apples does Emiily
Figure 7: The same proof tree Fig- 6 but with permuted ordering as in the experiments discussed in $3.4 In this specific example, the leaf node labeled with comp (I, J, 11, apple) is visited first (the other nodes follow post-order traversal). This translates into moving the sentence "Lucy has 11 more apples than John" to the front of the problem. As compared to the canonically ordered problem shown in Fig. 6 the sentence has moved two steps.
Layla has 13 fewer yellow plates than Sophia. The number of yellow plates that Emma has more than Henry is the same as the difference between the number of yellow plates that Amelia has compared to Sophia. What is the number of yellow plates that Henry has?
E.4 PERMUTED PROBLEMS
CANONICALLY ORDERED PROBLEM TEXT
Nicholas has 19 computers. Lucy has fewer computers than Nicholas. Harper has 6 fewer computers than Lucy. John has 10 more computers than Harper. Abigail has 15 fewer computers than John. Abigail has 12 fewer computers than Logan. How many computers does Logan have in their collection?
PERMUTED PROBLEM TEXT (DISTANCE) 1)
Lucy has 6 fewer computers than Nicholas. Nicholas has 19 computers. [Original Position] Harper has fewer computers than Lucy. John has more computers than Harper. Abigail has 15 fewer computers than John. Abigail has 12 fewer computers than Logan How many computers does Logan have their collection?
PERMUTED PROBLEM TEXT (DISTANCE 3)
John has 10 more computers than Harper. Nicholas has 19 computers. Lucy has 6 fewer computers than Nicholas. Harper has 6 fewer computers than Lucy. [Original Position] Abigail has 15 fewer computers than John. Abigail has 12 fewer computers than Logan How many computers does Logan have their collection?
PERMUTED PROBLEM TEXT (DISTANCE 5)
Abigail has 12 fewer computers than Logan. Nicholas has 19 computers. Lucy has 6 fewer computers than Nicholas. Harper has fewer computers than Lucy. John has 10 more computers than Harper. Abigail has 15 fewer computers than John. [Original Position] How many computers does Logan have in their collection?
21

---

Published as a conference paper at ICLR 2025
Nonlinear Problems
OpensAl (preview)
DeepSeek
1.0
Accuracy
0.0
Depth
Figure 8: Answer accuracies for generalization to increasing depth and width for nonlinear problems for OpenAI ol -preview and DeepSeek-R under zero-shot prompting and maximum context length of 4,096 tokens.
F ADDITIONAL ANALYSIS ON REASONING MODELS
Fig. 8 shows additional results the OpenAl o1-preview model ""ol-preview-2222-09-222 and DeepSeek-RI (DeepSeek-Al AT 2025). evaluated on the nonlinear test sets. We only used zero- shot prompting due to the high costs of performing inference on these models. Using simple, straightforward prompt is also consistent with OpenAl's current recommendations We see that these models show superior performance to all models from the main text, with ol generalizing better than R1. However, as with the other models, the performance decreases complexity increases.
Importantly, we noted that performance can be drastically improved by increasing the limit on the number of output tokens. The results presented in Fig. 8 used a token limit of 4,096, like the experiments in the main paper. For ol, we compared the effect of the token limit on an additional test set of 400 nonlinear problems with depth 7. With a token limit of 4,096, the model could only answer 0.25% of the problems correctly. However, with a token limit of 10,000, answered 76.5% of the problems correctly. For RI. the performance using a token limit of 10,000. These results are noteworthy and suggests that the inference techniques used by these models is highly effective for decomposing complex problems into smaller steps.
problems with depth 6 is increased to 77.5% when
To test the limits of these models, we performed a final evaluation on a subset of 100 of the depth 7 problems where the sentences in each problem call from 43.2 that permuting the sentences in random order is valid since all the inference rules used in our nonlinear problems both models, which is the current recommendation by OpenAI. On this set, the answer accuracy was only 5.0% for ol and 11.0% for RI. Thus, we conclude that MathGAP future- proof in the sense that it can generate problems on which current state-of soree-offttae ofetth-rrt reasoning models fail. We stress, however, that the aim of this paper not construct challenging test set for state-of ofte--ofthheart models; there are several other ways that MathGAP can be used to create problems that might be even more difficult.
randomly ordered (apart from the question). Re-
commutative. this experiment, we allowed 25,000 output tokens for
G LIMITATIONS
We did not prioritize linguistic diversity in our problem sets; future work may investigate how our findings generalize distributions of problems with higher linguistic diversity. We did also not consider or study the possibility of token biases (Jiang et al. 2024b). It is beyond the scope of the present study to consider problem texts languages other than English; however, is straightforward to extend MathGAP to other languages. In addition, we cannot guarantee that the distributions of our generated problems are different from those of any potential internal data gatta-ennrating mechanisms from Meta or OpenAI. However, we deem it to be unlikely that they use similar formalism as we do here, and it gets increasingly unlikely that problems are similar as complexity increases.
While the more complex nonlinear problems are indeed challenging even for the most capable models, we note the purpose of our study was not to create a challenging evaluation set for state-of-the-arr LLMs. If that is the goal, one could create harder problems by combining several inference rules (e.g., additionally including the rate concept; Opedal et al.] 2023), allowing a wider variety of proof-tree shapes, and presenting the axioms in arbitrary orderings. It is also possible create more syntactically complex templates (and paraphrase those using LLMs) and use numbers on which is more challenging for models perform arithmetic (Razeghi et al. 2022)
22

---

Published as a conference paper at ICLR 2025
Finally, it is hard to make direct comparison between the difficulty of linear and nonlinear problems, since they use different inference rules. In particular, nonlinear proofs contain comp-eq predicates, which are not present in linear proofs.
23


Reviews:
[{"review_id": "G2jRt0YPwB", "reviewer": "Reviewer_w9Zb", "summary": "This paper introduces MathGAP, a framework designed to evaluate large language models (LLMs) on mathematical word problems requiring proofs of arbitrary complexity. The authors address two critical issues in current evaluations: data contamination due to overlapping training and test sets, and benchmarks not reflecting the arbitrary complexity of problem proofs.\n\nMathGAP uses proof trees characterized by properties such as depth, width, linearity, and ordering, enabling the generation of synthetic problems with controlled complexity and corresponding chain-of-thought (CoT) reasoning annotations. By systematically varying these properties, the framework assesses how well various LLMs generalize to problems more complex than those encountered during training or provided in-context.\n\nThe authors conduct comprehensive experiments that reveal model performance significantly declines as proof complexity increases, particularly for nonlinear proofs. Interestingly, providing in-context examples from the same distribution as the test set does not always enhance performance. The study sheds light on the limitations of current LLMs in handling complex reasoning tasks and offers insights into their generalization capabilities.", "questions": "Have you considered incorporating paraphrasing techniques or using LLMs to generate more diverse linguistic expressions to increase variety and challenge the models? Incorporating more arithmetic relations, irrelevant information, larger numbers, or units to enrich tasks could help increase the diversity as well. \n\nHow do you ensure that the proof tree structures accurately reflect LLMs' reasoning processes? Is there a risk that models might not utilize the intended inference paths?\n\nCan you provide more detailed analyses to determine whether performance drops are due to arithmetic computation errors, logical reasoning mistakes, or other factors?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Addresses a Critical Gap: Tackles the underexplored area of evaluating LLMs on problems with arbitrary proof complexity. Directly addresses issues of data contamination and limited complexity in existing benchmarks.\n\nFormalism and Clarity: Utilizes proof trees and logical forms to precisely characterize problem complexity. Well-structured and clearly written, with helpful figures enhancing understanding.\n\nComprehensive Experiments: Conducts extensive experiments across multiple dimensions of proof complexity (depth, width, linearity, ordering). Evaluates various state-of-the-art LLMs, providing valuable insights into model behaviors and limitations.\n\nInsights into LLM Limitations and In-Context Learning: Reveals that model performance degrades with increasing complexity. Provides meaningful insights into the impact of different in-context learning strategies. Highlights the models' sensitivity to problem structure and ordering.", "weaknesses": "Synthetic Data Limitations and Linguistic Diversity: Synthetic problems may not capture the full linguistic and conceptual diversity of real-world problems. Reliance on templates could lead to repetitive linguistic patterns. \n\nAssumption of Reasoning Alignment: Assumes LLMs reason in ways closely aligned with the proposed proof trees. Models might use heuristics or patterns not captured by the formalism, affecting evaluation accuracy.\n\nLimited Error Analysis: Lacks a deep analysis of error types made by the models. More detailed error categorization could provide insights into reasoning limitations (e.g., arithmetic vs. logical errors).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WYuNl94iWM", "reviewer": "Reviewer_tyGU", "summary": "The paper introduces MathGAP, a framework for evaluating LLMs' mathematical abilities. The paper notes two main motivations for creating MathGAP: 1) data leakage/contamination and 2) lack of comprehensive and systematic evaluation signals (e.g., OOD generalization, different complexity levels, etc.). The authors show that the performance of almost all LLMs drops as the difficulty (measured by depth/width of proof tree) increases. Moreover, the paper shows that LLMs are sensitive to the order in which input assumptions/axioms are presented, suggesting limitations in the perceived reasoning capabilities of LLMs.", "questions": "See the questions from my comments above.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1- Given the unfortunate hype over LLMs, I believe it is crucial to have objective and solid evaluation metrics for LLMs and separate science from hype. This is exactly what this work is aiming to do.  \n2- The paper is extremely well written and the authors present the results very well.  \n3- Overall, the experiments are well-designed. However, they could have been extended to provide more insights (see below).", "weaknesses": "1- It is not clear how much \"new insight\" this work provides as many of the results have already been reported in other similar contexts. For instance, Dziri et al. also has a similar notion of graph depth/width, and shows that by increasing depth and width for both ID/OOD setups the performance drops significantly (see Fig. 5 of [1] for example). However, some open questions have not been addressed yet. For instance, from the results, it is not clear why the Llama3-70B model has a near-perfect score on \"linear depth\" scenarios: is it because of its training data, scale, or something else? Moreover, why is the performance gap between linear/non-linear benchmarks so significant? Do we know why the LLMs struggle so much in the nonlinear problems (e.g., Figure 3 of the paper) even when depth is small?\n\n2- I believe given the nature of MathGAP, the paper could have explored more fine-grained evaluation scenarios beyond the final accuracy with in-context prompts. For instance, according to the note in section 5, the number of shots is fixed to 12 and the footnote mentions that this is large enough according to another work. However, the impact of context on model performance on MathGAP is unclear: Does the number of shots have a similar impact on linear/non-linear setups? Why?\n\n3- The paper does not mention how large the overall set of proper names and values were. For instance, we know that LLMs have token bias and this can have an impact on results [2]. In addition, it is not clear what range of numbers was used and how many of the mistakes were arithmetic mistakes versus logical mistakes.\n\n4- The logical forms and inference rules used are relatively simple and may not capture more complex mathematical relationships. However, this is not a critical issue compared to other issues as we know that models still struggle in these simple cases as well.\n\nOverall, I think the work is borderline in its current form. I believe this work has potential to have a significant contribution if it can provide more insights and explanations that lead to better understanding of LLMs.\n\n[1] Dziri, Nouha, et al. \"Faith and fate: Limits of transformers on compositionality.\" Advances in Neural Information Processing Systems 36 (2024).\n[2] Jiang, Bowen, et al. \"A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners.\" arXiv preprint arXiv:2406.11050 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XgU4OKjgej", "reviewer": "Reviewer_d9We", "summary": "Paper presents a framework for systematically studying the OOD reasoning capabilities of LLMs by programmatically generating training and test examples. Paper also presents analysis of how existing models behave for MathGAP problems under different settings.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\n- Current LLM reasoning benchmarks are mostly comprised of fixed datasets, which might be susceptible to data leakage and do not offer a way change different characteristics of the datasets such as difficulty\n- MathGAP offers a way to systematically change the different aspects of the problem, size as depth, width and (non)linearity of problem tree, and constructs new problems every time, allowing for more controlled experiments on OOD generalization\n\nQuality/Clarity\n- Paper is well written and easy to understand\n\nSignificance\n- in addition to providing a new benchmark to study generalization, paper also conducts analysis of current LLMs behavior using the proposed benchmark\n- Presents interesting findings such as some models performing better when given OOD in-context examples rather than IID in-context examples", "weaknesses": "MathGAP examples seem to mostly involve a certain type of arithmetic problem, it would be interesting to expand to scope to include other types of reasoning problems\n\nIt would also be interesting to also study the finetuning generalization behavior of LLMs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QPmAxtVmO3", "reviewer": "Reviewer_iPba", "summary": "This paper proposes a math proof benchmark, called MATHGap, by constructing proof trees with predefined predicates and inference rules. Evaluation on a number of LLMs shows that LLM's performance decreases as the proof complexity increases. Many LLMs do not do well in these problems.", "questions": "I would appreciate if the authors could investigate the research questions raised in the 'weaknesses' section. In addition -\n\nWhy were different predicates used in different parts of the analysis, i.e. comp and transfer for depth analysis, part-whole for width analysis, etc.?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The authors did a good job elaborating the method. Section 3 provides a good background for readers to understand the method.\n2. Constructing out-of-domain math problems using the logical form is a good way to evaluate the math capabilities of LLMs.", "weaknesses": "1. Despite the many LLMs being tested, the paper seems to miss the latest state-of-the-art LLMs, including Llama 3.1, Llama 3.2 (since Llama 3.2 is only recently released, I understand that the authors did not have time to include it), and GPT-O1. This is rather important, because the linear problems were pretty well-resolved by GPT-4O, and Llama 3 is also already very competitive. Including the more recent models might change the conclusions. Therefore, the current conclusion seems a bit outdated.\n\n2. The authors seem to relate the failure of LLM's performance on the more complicated proofs to the OOD generalization failures. However, this is not necessarily the case because it may as well just be the performance degradation due to the problems being more challenging. In many cases, the in-distribution examples do not improve the performance over OOD examples, which is evidence that the performance drop is due to increased difficulty, not the generalization gap.\n\n3. The analysis in the paper looks a bit shallow. It does not show much insight into why and how the LLMs fail. For example, is the performance degradation as complexity increases simply due to the more proof steps? Namely, it may as well be the case that the error rate of each step is constant; it is just the increased number of steps that drive down the overall success rate. In this case, the model scales fine and it is just the success criterion that becomes more stringent. Also, what are the typical failure modes? Are the errors simple arithmetic errors, reasoning errors, or errors in parsing the final results? Why does the in-context example fail to improve the performance? Why is there a nonlinear relationship between the distance of the ordering and the performance? It would make the paper stronger if the authors could perform more experiments to answer these research questions.\n\n4. The MATHGap dataset looks too constrained for me. It only contains 5 predicates and 5 inference rules. As a result, the linear problems seem too easy for the latest state-of-the-art models. It would be nice to study how the performance scales with the number of predicates and inference rules, and whether the LLMs can retrieve the correct inference rules where there are many.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)