GSM-$\infty$: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Authors: Yang Zhou
Hongyi Liu
Zhuoming Chen
Yuandong Tian
Beidi Chen
Venue: ICML 2025 poster (2025)
Keywords: Long Context
Reasoning
Understanding
Benchmarks
Primary area: deep_learning->large_language_models
TL;DR: We study LLMs reasoning ability decay with respect to increasingly harder problems and with respect to increasing context length through synthesized dataset generator that generates fine-grainedly controlled GSM8K-like problems.

Abstract: Recently, long-context large language models (LLMs) have shown strong performance in information retrieval and long-document QA. However, to tackle the most challenging intellectual problems, LLMs must reason effectively in long and complex contexts (e.g., frontier mathematical research). Studying how LLMs handle increasing reasoning complexity and context length is essential, yet existing benchmarks lack a solid basis for quantitative evaluation. Inspired by the abstraction of GSM-8K problems as computational graphs—and the ability to introduce noise by adding unnecessary nodes and edges—we develop a grade-school math problem generator capable of producing arithmetic problems with infinite difficulty and context length under fine-grained control. Using our newly synthesized GSM-$\infty$ benchmark, we comprehensively evaluate existing LLMs. We find a consistent sigmoid decline in reasoning performance as complexity increases, along with a systematic inference scaling trend: exponentially increasing inference computation yields only linear performance gains. These findings underscore the fundamental limitations of current long-context LLMs and the key challenges in scaling reasoning capabilities. Our GSM-$\infty$ benchmark provides a scalable and controllable testbed for systematically studying and advancing LLM reasoning in long and complex contexts.

GSM-oo:
How Do your LLMs Behave over
Infinitely Increasing Reasoning Complexity and Context Length?
Yang Zhou
Hongyi Liu
Zhuoming Chen
Yuandong Tian
Beidi Chen
Abstract
Models Bubbles for Performance versus Price
Long-context large language models (LLMs) have recently shown strong performance in informa- tion retrieval and long-document QA. However, to tackle the most challenging intellectual prob- lems, LLMs must reason effectively in long and complex contexts (e.g.. frontier mathematical re- search). Studying how LLMS handle increasing reasoning complexity and context length is es- sential, yet existing benchmarks lack a solid ba- sis for quantitative evaluation. Inspired by the abstraction of GSM-8K problems as computa- tional graphs-and the ability introduce noise by adding unnecessary nodes and edges-we de- velop a grade-school math problem generator capable of producing arithmetic problems with infinite difficulty and context length under fine- grained control. Using our newly synthesized GSM-X0 benchmark, we comprehensively eval- existing LLMs. We find a consistent sig- moid decline in reasoning performance as prob- lem complexity increases, along with system- atic inference scaling trend: exponentially in- creasing inference computation yields only lin- car performance gains. These findings under- score the fundamental limitations current long- context LLMs and the key challenges in scal- ing reasoning capabilities. Our GSM-oo bench- mark provides scalable and controllable testbed for systematically studying and advancing LLM reasoning in long and complex contexts. Code open-sources at https: http://ffiiiiiiiiie github. io gsmm infinite/.
16k Context Performance
1200
Proo00002
Zero Noise Performance
900
Qwand-3-7280
Mistral Large Instruct-2411
Gerini-1 3-Flash-002
2ue
600
DeepSeok
Liame3.1 1-78 instruct
300
A2.5-7B Insseuntt
CP 20T-
Jantn
48-inknuur
0.1 Generation Price (USD per M Output Token) logscale
1
10
Figure 1. Evaluation of 10 powerful LLMs on GSM-OO, com- paring API generation cost (horizontal axis) with zero-context reasoning ability (vertical axis). Bubble size represents reasoning performance 16K context length.
performance a tremendously long context, where Team et al. (2024a) achieves near-perfect performance in 10M multimodal retrieval and long document QA. However, for long-context LLMs contribute cutting-edge mathemat- ical and scientific discoveries function as autonomous agents, they must be capable of processing dense, complex information and reason through multi-step tasks. For in- stance, Sir Andrew Wiles' proof (Wiles, 1995) of Fermat's Last Theorem 1995 spans more than 88K highly compact tokens with deep logical connections, making context-level RAG (Lewis et al., 2021) insufficient and highlighting the need for long-context LLMs. Therefore, it is erucial to benchmark and facilitate long-context LLMs for complex reasoning and high-density information processing.
Although widely used, current long-context benchmarks do not fully capture the true potential of long-contexl LLMs (Yu et al., 2024a; Li et al., 2024a;b), making it challenging to measure their progress toward advanced intellectual agents. It is mainly due the following three reasons: (1) Low Complexity. Many long-context benchmarks, such as I Long- Bench (Bai et al., 2025; 2024) and most tasks in RULER (Hsieh et al., 2024b), focus on retrieval or summarization, which involve low reasoning complexity. Similar to (Yu et al., 2024b), we found that simple context-level RAG
1. Introduction
Recently, state-of-the-art long-context LLMs (Team al., 2024a; MiniMax et al., 2025) have achieved astonishing
'Carnegie Mellon University "Meta GenAL. Correspondence to: Beidi Chen <beidic @ andrew.cmu.cdu>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

GSM-co: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Guen-3
GSM-Infinite
aAdult Lionatt Zon
MAdult Monkey st Zoo
Accurscy Zero. BK. 168, 32K Contest
Regime
nowbon Animall Children atfffforrion
Restening Come complant
Total: 7 ops
Average 4Neebom
Average Number of Newborn Coildren per Adult Monbny at 700
Children Adult lon Zoo
SOTA Capability
System: Assume any untmmaannt not murtieted dazari Lore.
GSMM-a
User: Mkdull Hirtoey al Jeftrron uquala a akouh at Exp seppeeeeee Aug. shoodde
Longllench vi.02
MMJUIl Hoxeoy 2oo, Whatl withe namter new sor cremet en forn al Dos?
Assistant Bag. thoolton Chiitren soitt Las uc sount + 2 (1) Aug. showlom
RULER
LOFT
Chidde per AGUET Moskes 2t 20s equate 2 i EII look nnw.om Acceral Chittron
2 18---11---111188 (1
Context Longth
ID
Figure 2. (a) We position existing benchmarks across the Reasoning complexity versus context length plot. Reasoning datasets are usually of very short context. Existing long context benchmarks are usually low in reasoning complexity. Our task can cover any context length that the user so chooses and can generate infinite reasoning complexity. However, for high reasoning complexity, our task needs to use a longer context for problems. Our task is shown in Red. (b) A simplified example of our datasct-building process. We first generate an interconnected computational graph, and we then based on the graph, attach real-world context to it to formulate the problem statements. (c) Shows Ow Instruct Score decay across zero-context, 8K, 16K, and 32K.
achieves on-par or even better results than long-context LLMs (shown in Figure 3). (2) Detectable Noise. Many tasks are innately short-context but are bloated into longer context through semantically irrelevant filler text (Kuratov et al. (2024) and crriable-tracing in Hsieh et al. (2024b)), which is easily distinguished by a retriever of context-level RAG. (3) Low Resource. While complex long-context tasks exist, such as long code completion (Loughridge et al., examples with ad- equate and verified annotation and labeling. This scareity limits test diversity and fine-grat fine-grained difficulty assessment, reducing their effectiveness in model evaluation.
from memorization and ensuring a clear focus on reasoning.
We introduce GSM-20, a long-context benchmarking framework that scales and controls reasoning complexity and noise through fine- fine-grained manipulation of computa- tional graphs, enabling their translation into diverse, human- and L-M--dedeble problems (Figure 2(b)). Specifically,
in Section 3.1, 3.2, we discuss how to control and scale reasoning complexity by manipulating computa- tional graphs and how to insert noise.
2024), they lack sufficient
Section 4.1 discusses methods ensure comprehensive coverage of reasoning patterns appeared in GSM.8KK
Ideally, a long-context reasoning benchmark should (1) offer controllable and scalable complexity, (2) incorporate hard- to-distinguish noise, and (3) support infinite data genera- tion for continuous and adaptable evaluation. Inspired by Delétang et al. (2023); Ye et al. (2024a), we model reasoning problems as computational graphs attached with language semantics. By adjusting their structure and complexity, we gain finengrained control over reasoning difficulty and en- able infinite scaling. Instead of inserting semantically irrele- vant filler text, noise is introduced as additional nodes and edges upon the core graph, strategically connected to exist- ing nodes without affecting to the necessary reasoning steps for solving the tasks. This design enables the generation of arbitrarily long test examples, difficult to differentiate relevant information from noise by context-level RAGS.
Section 4.2 shows ways enabling the computational graphs to LLM-understandable natural language mapping.
Figure 2(a) shows where our benchmark positions among existing benchmarks, demonstrating the effectiveness of GSM-oo for long-context reasoning evaluations.
We conduct comprehensive evaluation of 17 state-of-the- art LLMs on zero-noise problems and 10 LLMS on various noise-injected tasks using GSM-c0. We inclusively cov- ered a wide range of models, including both popular closed- source and open-source options, conventional transformers and hybrid architectures, models of varying sizes, and both reasoning and non-reasoning LLMs. In zero-noise settings, recent maasoning-optmmizen LLMs demonstrate substantial improvements over their non-reasoning counterparts. No- tably, Deepseek-R1 (DeepSeek-Al) et al., 2025) achieves an average AUC score nearly four times higher than previous SOTA models. However, in noise-injected scenarios, LLMs exhibit varying degrees of performance degradation. Our analysis reveals several key observations:
However, several technical challenges must be addressed to construct practical benchmark. First, ensuring diverse reasoning patterns and operations- - ideally providing a com- prehensive coverage of GSM-8K crucial to enable fair and thorough evaluation. Second, computational graphs must be effectively translated into natural language that is both human- and LLM-readable climinating ambiguity
Decay with reasoning complexity: LLM performance follows a strikingly consistent sigmoid or exponential
2

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Lia
and RAG on Long Context Tasks
100
100.0 100 102.0
100.6 100.0
sa...
Je.0 100.0 100.0
100.0
90.2
LLM
75
32.0
74.0
OnePassRAG
62.0
56.0
InteractiveRAG
50
2811
31.2
25
RULER
niah_
niah mk
riah mv
riah mg
Hard
Long LongBenchv2
120
80
40
+2.00
23.17
22:49 20:1
23.22
25
2:00
717
LongBench multinews guv_reeporl
casper
triviaqa
repobench-p
ArguAna
FIQA
Quora
Quest
LOFT
Figure 3. Study of Llaam3..117000 1-70B-Instruce with Passive RAG (referred to as OnePassRAG) and Active RAG (referred to as InteractiveRAG) on popular long-context benchmarks: RULER (at 64K context length), LongBench (>-8), LongRenchV2, and LOFT (128K context length). RAG is under the 2048 retrieved token hudget, and the decoder used for the RAG is Llama-3. I-ntB-Instruct. RAGs generally have robust performance, on par with the corresponding LLMs, showing that previous long-context benchmarks are either too simple reasoning complexity or contain detectable noise.
decay problem difficulty increases, highlighting funda- mental limitations in scaling reasoning capabilities.
ationship Values Across Diffe and Semanitic Hierarc hies
Number ope retorm GSMMIK set
Deskot Ope Dnly
Twx: entry Vartatles Max.
Decay with noise: Performance degradation intensifies as context length increases within the same difficulty level, while longer context brings sharper degradation of the LLM performance.
300
LLM thinking patterns: Models consistently perform better on forward-thinking tasks than on backward- thinking ones, suggesting a fundamental asymmetry in their reasoning strategies.
Rp6
(a)
(b)
Influence of Repeated sampling: Repeated sam- pling (Brown et al., 2024) on GSM-co reveals a clear pattern: performance improves linearly with increased inference steps but at an exponentially growing compu- tation cost, underscoring inefficiencies current I.I.M inference strategies.
Figure 4. presents conservative estimate for each problem difficulty in GSM-8K 8K 1.3K test set. We evaluate the difficulty of the problems by number operations needed get the final answer. The op count ranges from 2 to 12, while most are around 3-4. (b) shows Llama3. --B-Inrtrctt performance across differ- ent semanties hicrarchies, revealing the hidden reasoning difficulty innate in natural language.
2. Related Work and Problem Statement
Despite the wide popularity of some existing long-context benchmarks, this section reveals and elaborates on the three key limitations: low complexity, detectable noise, and low resource or limited quantity of test examples. These three limitations make extremely challenging to measure long- context LLMs' progress toward advanced intellectual agents using existing benchmarks.
As shown in Figure 3. our experiments demonstrate that RAG systems achieve competitive results with Llama3. 1- 70B-Instruct across these datasets. Notably, RAG out- performs L.MSs retrieval-focused tasks (e.g., RULER, LOFT) and performs comparably in text summarization and QA (most LongBench tasks), well as structured reason- ing problems such as variable tracking (RULER-vt) and code completion (LongBeecch-repobecchh RAG provide strong baseline while being substantially more efficient.
Low Complexity. A significant portion of long-context evaluation datasets, including RULER (Hsich et al., 2024b), LongBench (Bai et al., 2024), LongBench v2 (Bai et al., 2025), and LOFT (Lee et al.. 2024), primarily assess re- trieval and summarization rather than complex reasoning.
Detectable Noise. Noise consists of irrelevant details that divert the model from the core task, Many long-context
3

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Liama-3. 1-7---inrrrttt us RAGS (Medium)
Lierna 3.1 708 Instruct vs RAGs (Hlrdd)
Disance to QMEy
Lama 3.1 70B Instruct
Uama-5 1. 1-708- no8-nnstruet
RAG
RAG
0.5
- Interactiive RAG
percent threshold
Inturactoon BAG
threshold
KCCL ACC.recy
pertentiu (a) BE Medium
0.4
0.-
0.8
00
17%
00
5.0
00.0
15.9 17.
Percentle (b)
(c)
(d)
proposed long- long-context benchmarks. (a) studies retriever's behavior on the first 100 chunks of random problem in vt from RULER with 8192 context length. The chunks that need to be retrieved to solve the problem are labeled in coral, while the noise in blue. The chunks have retriever scores ranked from large (semantically far) to small (semantically close). Retriever locates the essential chunks with high precision, classitying all necessary chunks with the right side of the spectrum: (b) contrasts vt with our long-context benchmarks, showing that the retriever cannot locate precisely which chunk retrieve. (c) and (d) display the performance of two RAG systems on our benchmark medium and hard tasks. (Figure best viewed in color)
Figure 5. RAG performance
benchmarks artificially extend short-context tasks by injecting extraneous filler text (referred as noise or dis- tractor) that does not contribute to solving the prob- lem, allowing retrieval-based models to filter out noise effectively. In RULER's 's variable-tracing task with an 8192- token context, Llama3. 1-7BB Instruct achieves 100%, while OnePassRAG and InteractiveRAG reach 82.4% and 98.4%, respectively, despite using only 2048-token retrieval bud- get. A detailed breakdown in Figure 5 (a) reveals that retriev- ers consistently identify and prioritize relevant information while disregarding injected noise. These findings indicate that existing long-context benchmarks do not adequately justify the need for expensive long-context LLMs, as RAG systems effectively mitigate the noise impact and achieve similar performance.
3. Computational Graphs
In this section, we detailed the construction of computa- tional graphs, the key construct that enables GSM-oo gen- eration of infinite quantities of arbitrary context length and reasoning complexity. Specifically, we cxplain in detail the potential mapping between reasoning problems and com- putational graphs in Section 3. and in Section 3.2, we propose to generate indistinguishable noise by strategically extending the computational graph.
3.1. Graph Construction to Build Reasoning Problems
After carefully studying GSM-8K problems, we draw the following crucial observations, which allow us map ran- domly generated computational graph to level math reasoning problems that cover all possible operations and relationship types.
Low Resource. Many high-quality reasoning tasks (Math and coding) heavily rely on human annotation and labeling and have test examples in limited quantity. Here we use GSM-8K (Cobbe et al., 2021a) a) as an example, shown in Figure 4(a). It is infeasible to extract subsets of examples with exact op 8 for precise LLM evaluation due to the limited number of available cases only 26 in total, with even fewer satisfying op 2 8. This scareity makes meaning- ful evaluation impractical. Also, DafnyBench (Loughridge et al., 2024), a high-quality long-context coding benchmark only contains 782 verified and deduped examples.
Mapping Explicit Ops to Computational Graphs From Every operation used in the GSM-8K is one of the four "+", " "x", and " Consider the following example when op- erations are presented explicitly, "Eggs cost twice as much as tomatoes, while tomatoes cost 1 dollar each." These state- ments mention operations ("plus", "more", "times", etc.) can easily be abstracted out as a computational graph with variables, "dollar per egg" and "dollar per tomato"", as nodes. There are two edges one pointing from "dollar per tomato" to "dollar per egg", while another one from a constant 2 to "dollar per tomato". Another similar example is shown in Figure 6 (a). Therefore. randomly generating a computa- tional graph with different topology of edge connections will lead to a new reasoning problem once the natural language context are attached to the nodes of the graph.
Problem Statement How can we develop a benchmark that contains sufficient problems at every fiteigrained level of reasoning difficulty, from easy retrieval tasks to infinitely hard challenges. while providing infinitely customizable context length with high information density?
4

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Questions Contains Only Two-entity Variables Induces Implicit Operations and-
Questions Containing Three-entity Variables Additionally Induces Implicit oparations x and
#Lionn in
Arimal Irow Location *Morkoysi Zoo"
#Licns Zoo
ona Zoo
Aentemmnnm A Asinaiipe Location /hnnddoo Cit liiing pa hourt
VDoga Park
APigs Park
#leedem Clapher
Tote Newebo Ara Chidden
Q: Diven't, [3) equets 5 [1] [5] equats sum [f and [5] lo?
HHonkeyiin Yroel
Ober and 22), Ahark ESANT FaatOO Conoor www.aaanntttaaaaas..o Poot
Fotall Antmaain in Part
Q: Givvn FRI, what ia me iceat number of aniinata in 2007
TetatArimatin 20g BLaona in 200 Mifonness in 208, (Porword)
Chaccen ALLI JEB, (anwnrd)
Winng in
C Olvan (1): [a] (1) What la [3)?
PMontoya Zodi TecalArimaiin Zeg pilloo 200, (Reverse!
Qg: Ghean F13, (S1, what the Numberol of Moniays 2007
internen Zoa- Blonn nn Waverns
Whieemam Cn in par Linn Toral Nesttrr werttttrrnnee Awtraannnnnn.... Chriannt
(b)
(c)
(d)
Figure 6. Computational Graphs Illustration. (a) shows a simple computational graph, where every node and edge can be straightforwardly converted to statement, the other way conversion works as well. (b) breaks down the essence of implicit operations and the abstract parameter constructions; here, an assumption provided in the system prompt is to assume the animals not mentioned in question don't ' exist; essentially, problems in natural language omit the two red arrows in the computational graph. (c) provides an example problem when all variables that appeared are "two-entity" variahles, where only implicit addition/subbraation can he generated. (d) contrasts (c) and shows an example that with additional "three-entity" variables, the computation graph can also generate implicit multiplication/diiisioo. Both (c) and (d) also illustrate the design of reverse mode that specifically aims to generate implicit subtraction and division. (Figure best viewed color)
Generating Implicit + using Computational Graphs - On the other hand, the operations can also be presented implicitly hidden in natural language hierarchies. "Mary earns 20 dollars in the morning, while she earns 25 dollars in the aftermoon. How much total she earned that day?" Although the problem doesn't it explicitly mention addition, the solution has to sum up 20 and 25 to get 45. The reason is that natural language assumes a working day consists of morning and aftermoon. Similarly, all four operations can be hidden in natural language hierarchies. Inspired by Ye et al. (2024a), we adopt its construct of "Abstract Parameters" and "Instance Parameters" to construct computational graphs that facilitate the generation of problem statements containing the hidden operations. Essentially, the newly added con- structs can be thought of as adding the "total money" as a new node to the computational graph, which has two edges coming in, one from node "Morning money" and the other one from node "Afternoon money". But when generating the problem, we omit the description of two edges pointing to the node "total money on Friday". We also illustrate it in Figure 6(b). Following (Ye et al., 2024a), we connect the specific instance parameters to the abstract parameters using red edges. To reiterate, these edges are omitted, inducing implicit operations solvable only by LLM commonsense.
generate implicit operations of + and will be induced, but not x and +. For a problem contain implicit operations x, problems must contain variables with more than two entities in its name. For example, "Mary works 8 hours on Friday. Her hourly rate on Friday is 10 dollars. How much she will earn on Friday in total?" The variable "money per hour" contains "money", "hour", and "Friday" three entities, where "money" is an attribute of "hour", while "hour" also an attribute of "Friday". We further contrast the above scenario in Figure 6(c) and (d), we that if every variable is two-entity or "Animal" in "Location", generating x isn't straightforward, but once we add in the third entity, or an attribute of animal, generation of x becomes natural.
Scaling up Reasoning Complexity and maintaining con- trol with Computational Graph - Our computational graph generator employs the abstract parameter construct to gener- ate implicit operations and three-entity variables to represent multiplication operations. During the generation process of problem, specifically, a query node is first sampled from the graph. The corresponding topological sort list-ending with the ing as a measure of the problem's 's reasoning complexity. This approach enables the generation of a vast number of synthetic graphs. Furthermore, adjusting the number of variables provides coarse control over complexity, which is refined through precise filtering to produce well-defined subsets of graphs that meet specific operation constraints.
the shortest solution path, serv-
Generating Implicit x using Computational Graphs every variables in the above mentineet problem with the implicit "+" operation are "two-entity variables". "Morning Money" contains two entities, "Morning" and "Money", where in the context, "Money" is an attribute of "Morning". Same with "After- noon Money". In fact, out of all the examples we manually examined in GSM-8K, the minimum number of entities in the variable name is two, However, if every variable in the problem are two-entity" variables, only
3.2. Noise Construction Using Computational Graphs
Spider Topology - We observe that we can view noise as extending the computational graph to incorporate fake and unnecessary parameters and operators. However, two
5

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
instance parameter design is that it is only able generate problems with solutions with the "forward" and construc- tive ordering. Shown in Figure 6 (a) and (b), the design dictates that the specific and detailed variables should be defined before a more abstract variable. For example, "the number of Lions in Zoo" and "the number of Monkeys in Zoo" have to be defined before "Total Animal in Zoo" defined. The "forward" ordering leads the inability to generate implicit '-' operations for 2-entity problems and implicit "+" operations for 3-entity problems that require the more abstract variables, e.g. "Total Animal in Zoo", to be defined before a more specific variable, e.g. "the number of Monkeys in Zoo".
Noise Variables
Essential Variables and Logical Connections
Noise Variables
Figure 7. The Illustration of Noise as Extension of Core Computa- tional Graph.
critical questions emerge. First, how to extend the compu- tational graph without contaminating the original graph's ' solution and contaminations? We found that edges have to point outwards from the nodes in the original graph to the newly added noise nodes, essentially preventing the noise nodes from contributing to the core graph. Second, how to maximize the chance that RAG cannot retrieve the essential graph? It turns out interconnecting edges between newly added noise nodes won't 't contribute help detering RAG's retriever. We find out simple trick works well: ensuring the majority of the added edges connect core nodes and the noise nodes contribute to semantically close noise. We call this design Spider Topology as shown in Figure 6(c).
To generate all four kinds of implicit operations, we introduce "reverse mode" to generate the computation graph. Essentially, the graph construction still continues as before: starting with specific detailed variables and growing to incorporate more abstract variables. When it completes and we know all the values of nodes in the graph, we then randomly mask out specific initial low-level variables and force the solution to traverse the reverse direction as in the "forward" ordering. We present the illustration of data generation in Figures 6 (a) and (b) for the 2-entity and 3-entity, respectively.
4.2. Challenge 2: How to Ensure L.L.-mnderstandable Problem Generation?
We evaluated the resulting noise using two RAG systems presented in 2. The results are shown in Figures 5(c) and (d). Llama3.1-700 1-770B-Instttet achieves drastically stronger performance than the two RAG systems on 8K 2-entity problems and 3-entity problems. We also carried out the same study before on our data set setting 2, in Figure 5(b). We found that the RAG retriever now completely cannot distinguish which essential chunks from noise chunks, showing a clear contrast with yt tasks of the same context length in (a).
Mapping computation graphs to natural language is critical for evaluating LLMS' reasoning capabilities. To automate this process, we develop inter-swappable templates that enhance linguistic diversity while maintaining clarity. Several key considerations inform our design.
First, Certain syntactic forms, such as possessive construc- tions (e.g., A's B), are straightforward to encode but mis- lead LLMs due to their deviation from natural language. For example, South Zoo's 's Penguin is restructured as Penguin in South Zoo, and South Zoo's 's Adult Penguin's 's Average Num- ber of Newborn Children becomes Average Number of New- born Children per Adult Penguin in South Zoo. Through extended trial and error, variable names with different entity numbers are presented naturally in all prepared templates.
4. GSM-c0
In this section, we present key techniques that enable the synthetic dataset to be diverse in operations in Section 4. 1, LLL-understandaalle and enable the evaluation to be free from non-reasoning factors in Section 4.2. Then, we present synthetic problem generators capable of generating grade- school math questions with arbitrary reasoning difficulty and context length. Thus, we generate a suite of benchmarks called GSM-X detailed in Section 4.3.
Second, the number of constraints applied to the random graph generation process is as little as possible, forcing templates to enforce unit consistency across two-entity and three-entity variables to enable assignment between these two. For instance, "The average number of animal children per penguin in South Zoo" must share unit with "The number of penguins in South Zoo" to allow variable assignments.
4.1. Challenge I: How to Generate Implicit - and Operations?
Firstly, we review why the abstract-instance construct can only generate "4" and x but not "_" and +. The key limitation of Ye et al. (2024a) abstract parameters and
Third, to ensure real-world knowledge doesn't 't confuse the LLM's decision, we avoid specific real-world locations,
6

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
people's names, and festival names from appearing in the template. Based on these constricts, we propose three different templates that meet real-world templates: children- animal-zoo, teachers-school- -istrii,,,, and awards-movied festival. We present in Appendix G an ablation study show- ing that three templates are consistent in overall performance with only minor fluctuations when evaluated using Llama- -.n-tu-Inttruct. At each op, equal problems are tested for both constructive ordering (forward) and reverse ordering (reverse).
Fi: 2 2-65-nuttcc
reppared waritt trale Dng (Cg)
[a]
(b)
4.3. Benchmark Details
With the synthetic problem generators detailed in Section 4, we then use them to generate problems to build suite of reasoning tasks with increasing complexity. For the brevity of reference, we refer to the generated problems with only explicit operations as "Easy", the generated problems with 2-entity variables at maximum as "Medium", and the gener- ated problems with 3-entity variables at maximum as "Hard.
Figure 8. (a) shows repeated sampling on zero-context Hard task with Qwen-2.5-7B -.ee-..--7-Innstrrutt (b) shows the AUC to repeated sam- pling number of trials. We show that for repeated sampling, ex- ponentially increasing inference compute only leads linear increase in AUC improvement.
5. Evaluation
Ideally, when evaluating an LLM, we want to evaluate all difficulty levels, from the most basic logic complexity to when it completely fails to solve any problem. For the Easy subset of problems, it usually leads to large operation counts for powerful LLMs. However, although complexity-we not challenging, L.Ms trained with internal COT tend to generate very long arguments, saturating their API output generation limit (4K for many models). Thus, we observe sudden decay in accuracy in large ops, not because of LLMs' ability bottlenecks, but because of the above-mentioned nuance. Thus, we make a tweak to its problem: Instead of
In this section, we present comprehensive evaluations of various LLMs on GSM-20. Specifically, the section is organized as follows:
Section 5.1 presents the complete comparison of LLMs on both zero-noise tasks and long-context tasks. Besides the main leaderboard, we further share four interesting findings.
Section 5.2 reveals that the sigmoid function generally fits the L.LM performance degradation to the increasing ops well.
Section 5.3 shows that LLMs generally and consistently perform forward problems better than reverse problems. (Defined in 4.1)
asking the LLM to find the value of one variable, we ask the
LLM to find all the variables that have some value specified, effectively increasing the difficulty of the problem.
Section 5.4 discuss that various LLM performance decay over longer context and further ablation of the noise. Section 5.5 shows that GSM-oo, exponential increase in inference compute yields linear AUC gains.
For the modified Easy subset, we keep the generated prob- lem in the most basic form: symbolic assignment. The typical problem statement then becomes "v1235 equals v1468 plus 1." Since the modified problem is not easier com- pared to Medium and Hard, we now call it "Symbolic". For Medium and Hard, we used all three templates and mixed the generated problems together to ensure diversity. For reporting L.LM performance, we use Area Under Curve (AUC), shown in Figure 10, which is computing Rie- mann sum over the LLM's performance in accuracy versus the number of operations from 2 to when its performance is lower than 5%.
5.1. Leaderboard
We evaluated 18 powerful LLMs on zero-noise problems, resulting in Table 1, while 10 models are evaluated on the long context as shown in Table 2. From Table 1, we can see that the score separates these LLMs into clear groups Reasoning models (R1 and o1-mini) are significantly ahead of the rest of non-reasoning LLMs. On the other hand, models with hybrid architecture aren't performing strong on the zero-noise pure reasoning benchmarks. MiniMax- Text-01 and Jamba both severely underperform compared to 70 Bevel models. Also, similar things can be discussed when comparing 70B and 7B level models. In Table 2, we see that the models show a very different decay pattern, while Deminni-15-PPre is significantly ahead of the rest of the models, Reasoning model evaluation remains too costly
We prepare zero-noise, 8K, 16K, and 32K in the benchmarks. The existing generation pipeline is capable of generating in 16M context, but the smaller 70B level models effectively failed in the 32K context already, while evaluating larger ones brings cost beyond our acceptance.
7

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Table 18 selected models are evaluated GSM-OO zero-noise henchmarks using Area-Under-Curve (AUC), which is computed by taking the Riemann Sum of accuracy versus op count from 2 to when the model accuracy drops below 5%. We also present detailed statics of the first op number the model to have an accuracy lower than 50%, 10%. and the average accuracy the first 30 ops settings. Besides, also highlight the reasoning models, linear attention hybrid models, and SSM hybrid models. Due to space and "Claude-3.5-Haiee has version number 20241022; "GPT-40-2024- 11-20" is shortened as "GPT-40" and "GPT-4o-mii minimmiini--------- is shortened as "GPT-4o-mini",
constraint, "Nistra-largge-sssuutt-41111"1
shortened as "Mistral-Large": " "CClaude-3.5-Sone
Three Subtasks
Detailed Statistics on Hard Subtask
Score
Models
Symbolic
Medium
Hard
1st op
1st 10%
Avg. Ace op
Avg.t
DeepScck-RI
7280.0
9750.85
8573.8
100
> >10
0.9427
8534.88
GPT-o3-mini
6690.0
8335.66
5769.96
70
110
0.9423
6931.88
GPT-ol-mini
5060.0
6054.91
3738,43
50
90
0.8397
4951.11
DeepSeek-V3
4310.0
4100.81
2407.86
24
0.6669
3606.22
3530.0
3205.75
1846.19
21
50
0.5403
2860.65
2547.0
3659.59
2318.28
26
45
0.6924
2841.62
Claude-3.5-Sonne
2161.0
3281.8
2115.79
26
40
0.6758
2519.53
Mistral-Large
2332.5
2879.92
2310.49
24
0.6645
2507.64
Owen2.5572B-Insttuul
2048.0
2496.81
2016.38
21
0.5433
2187.06
GPT-4o
2379.0
2457.37
1451.54
18
30
0.5064
2095.97
Bemini-1.5--laas-000
1970.0
1478.75
1274.25
13
30
0.4460
1574.33
Llama3. 1-70BB--Intrruue
1769.0
1650.25
1205.25
15
30
0.4314
1541.50
MaxiMMa--Tex--O
1618.5
1712.64
1178.51
0.4213
1503.22
GPT-4o-mini
1389.0
1406.5
913.89
12
22
0,3094
1236.46
Claude-3.c -Gaud-3355-aaik 3.5-Haiku
897.0
1053.16
784.34
10
22
0.2910
911.50
Breen2557B-Insteeee
786.95
886.75
618.5
7
19
0.2257
764.07
Llama3.1 1-Stttntttttt
462.0
786.5
606.5
6
0.2186
618.30
Jamba- 1.5-Large
856.0
485.13
466.4
0.1828
602.51
prisingly, most models exhibit sigmoid-like performance decay, as shown in for forward problems (see Section 4.1 for definitions).
Table 2. 10 selected models are evaluated on GSM-OC Long Con- text benchmarks using Average AUC of Symbolic. Medium, and Hard. We evaluated models on 8K. 16K. and 32K context. Al- though our pipeline is capable generating longer problems, the resource required to go further for larger models beyond our ceptance, while smaller models effectively has completely failed.
Under the Medium subtask, LLM performance aligns re- markably well with sigmoid function curve, with R2 0.98. The pattern is intuitive: at low operation counts, accu- racy remains near 1.0; complexity increases, performance first decays gradually, then drops sharply toward zero, where
Model
8K
16K
32K
Avg.t
Gemini-1.5-PP P.5-Pro-022
1182.43
896.31
812.96
963.9
927.33
681.53
563.65
724.17
L.MSs effectively fail to solve the problems. The score even- tually stabilizes near zero. Interestingly, while LLMs vary in overall capability, they follow the same trend, differing primarily in the decay rate of the sigmoid function.
Mistral-Large
914.49
563.73
319.21
599.14
DeepSeek-V3
935.10
477.02
313.66
575.2
Gemini-1..5Flash-0002
673.88
476.72
377.38
509.3
Llama3.1-70p 1-BBB Instruct
479.00
394.50
355.5
409.67
Mext-Ox
481.32
359.56
325.95
388.94
Go---o-mini
401.00
337.81
275.63
338.15
5.3.
Reverse Problems are Harder Solve for LLMs
Swwe2.55-7BIInstruu
248.00
211.50
196.17
218.56
1aama..1-8B-Inseie
183.67
149.50
109.45
147.54
The generator of GSM-o0 can generate both the "forward" and the "reverse" problems, which can be compared sep- arately. Most LLMS perform worse in reverse problems than forward ones, shown in Figure 9(b) using Mistral- Large (Jiang et al., 2023a). For the Medium, only Jamba- 1.5-Larre out of a total of 18 L.MMs evaluated do reverse problems better than forward problems, and the average difference AUC is 604. For the Hard subtask, 5 out of 18 models have larger reverse problems AUC, Deepseek 2.ren2.5-7B-Instruue (Qwen et al., 2025), 4o-mini, and Jamba- 1.5-Lage (Team et al., 2024b), The average difference in AUC is 154. A detailed break- down is listed in Appendix F. Besides, LLM performance
or slow for current long-context leaderboard participation.
Later on, examine the behavior of LLMs on GSM-c0 and summarize four interesting findings uniquely enabled by our method of problem generation.
5.2. LLM Performance Degradation Can be Modeled Using Sigmoid Function
R1, Ge5mini11.5-Pro-000
Our construction of GSM-oC enables precise measurement of LLM performance across ine-grained difficulty levels. For each subtask, we observe a clear trend: LLM accuracy declines as the number of required operations increases. Sur-
8

---

GSM-co: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Slgmold fit Zero context Medium Forwerd Data
Miseral Zere th Forwerd and Reverse Separation
Lleme- 3. 10000trrstt on different noises
systems different noges
20
s3-etet
inteiia Rardam
GSM ofrina Ganeraee and Randon
ase 1M c3 mirl
0.3 2.3 728 iestrvcd
Orean 728 instrct
Revana only
Revame only
fowwrd
03
0.10
Q.wn 2at.:2.-----ttt-e Iettret
Accurccy
Accurnccy
12,00
7,66
R*2 0.9625
62
R-2-0.0022
IP2+ 0.95431
op count
Up count
25
57
25
300 12.4
15. 315
28
COunt
(8)
(b)
(c)
Figure 9. (a) shows three different L.MSs behavior on GSM-0O henchmark zero-context Medium Forward, GPT ol-mini, Qwen2.5- 12B--nnsruct, and sigmoid well, all with R² >0.98. (b) shows the gap between forward-thing thinking and reverse- -hinnssitiiigggggg problems from Mistral Large on zero-context Hard. reverse-thinking problems are significantly harder and can approximated by the sigmoid function that is essentially left-ward shifting from the forward sigmoid function. (c) and (d) presents RAG and corresponding LLMs' performance on different noises. Other than ours, RAG even improves performance.
These models are drastically different in reasoning ability but have performance be modeled by
on reverse problems can also be modeled by sigmoid mapping. More plots are presented in Appendix F.
Surprisingly, suppose we calculate the AUC score under every curve corresponding to each number of repeated trial settings. In that case, we find that the increment in the AUC score from two consecutive settings is close. If we plot the AUC score versus the number of repeated trial settings and take the log scale of the repeated trial N, the graph is linear, as shown in Figure 8(b). The R-squared is greater than 0.99 for both Swen2.5-72B-Instnmue and 1-8 where Llama3. 1..------Inntc... Therefore, GSM-co helps reveal that Repeated Sampling gives linear AUC Im- provement from exponentially increasing compute cost.
5.4. Long-context Degradation and Noise Ablation
We evaluate LLM performance across increasing context lengths (0, 8K, 16K, 32K) and observe a consistent decline in performance as context length increases. Notably, models exhibit different decay patterns. We present results for 10 models across subtasks, each with four curves representing different context lengths. All 30 plots are in Appendix I..
We conduct an ablation study three noise types: GSM- 0o (ours), LLM-generated, and random. For LLM- generated noise, we prompt GPT-4o to create a fake documentary-style commentary on random problems, oc- casionally introducing nonsensical variable mentions. For random noise, we follow Hsich al. (2024), using generic statements like "The sky is blue. The tree is green." We and RAG system under
6. Conclusion
Long-context L.LMs have the potential to tackle complex, information-dense tasks requiring deep reasoning and co- herent long-form generation. To advance their develop- ment and benchmarking, we introduce GSM-o0, a syn- thetic long-context reasoning benchmark generated entirely by software-based system with fine-grained control over complexity and information density. Through extensive evaluations GSM-oo, uncover key insights to inform future LLM training and inference improvements.
evaluate Llama3.117000 all three noise types in an 8K context. Interestingly, RAG outperforms long-context LLMs on LLM-generated and ran- dom noise, effectively filtering irrelevant content. However, fails to distinguish GSM-00 noise from crucial sentences.
5.5. Limitations of Repeated Sampling
Impact Statement
The construction of GSM-00 allows us to study techniques for Inference Scaling as well. Specifically, we study re- peated sampling (Brown et al., 2024; Snell et al., 2024) and its effectiveness under different reasoning complexity. We study both Qwen2.5-7B Instruct and Llama3. 1-0-3.1--B--nnttue with the best-of-N settings similar to Brown et al. (2024). Interestingly, we find that repeated sampling seems to boost the performance the most for smaller op count subsets, and the benefit of repeated sampling diminishes gradually for larger op count subsets, Qwen2.5-7B 7-n-struttt behavior is plotted in Figure 8(a) with different repeated trial settings.
This paper presents work aimed at advancing the field of Ma- chine Learning, specifically in long-context reasoning for large language models. Our benchmark, GSMoo, provides scalable and controllable testbed for evaluating and im- proving LLM reasoning capabilities in complex, extended contexts. While this research contributes to the broader development of AI reasoning, we do not foresee any im- mediate ethical concerns or societal implications beyond standard considerations in Al research, none of which we feel must be specifically highlighted here.
9

---

Q. Du, Q. Ge, R., Zhang, R., Pan, R., Wang. R., Chen, R. J., Jin, R. L.., Chen, R., Lu, S., Zhou, S., Chen, S., Ye, S. Wang, S., Yu, S., Zhou, S., Pan, S., Li, S. S., Zhou, S. Wu, S., Ye, S., Yun, T., Pei, T., Sun, T., Wang. T., Zeng, W., Zhao, W. Liu, W. Liang, W., Gao, W., Yu, W., Zhang. W., Xiao, W. L., An, W., Liu, X., Wang, X., Chen, X., Nie, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yang, X., X., Su, X., Lin, X., Li, X. Q. Jin, X., Shen, X.. Chen, X., Sun, X., Wang, X., Song, X., Zhou, X., Wang, X., Shan, X., Li, Y. K., Wang, Y. Q. Wei, X., Zhang, Y., Xu, Y., Li, Y., Zhao, Y., Sun, Y., Wang, Y. Yu, Y., Zhang, Y., Shi, Y., Xiong, Y., He, Y., Piao. Y., Wang, Y., Tan, Y. Ma, Y., Liu, Y., Guo, Y. Ou, Y., Wang. Y., Gong, GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
References
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang. Z., Du, Z., Liu, X., Zeng, A.. Hou, L., Dong, Y., Tang, J.,
and Li, J. Longbench: A bilingual, multitask benchmark for long context understanding, 2024. URL https: //arxiv... org/ abaaaaa///20 2308 14508.
Bai, Y., Tu, S., Zhang, J., Peng, H., Wang, X., Lv, X., Cao. S., Xu, J., Hou, L.,, Dong, Y., Tang, J., and Li, J. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks, 2025. URL https: / /aa/i.iiivvv/.. www.rxvv.....aaa.. org/abs/ abs//as///22 2412. 242241
Beltagy. I., Peters, M. E., and Cohan, A. Longformer:
Y. Zou, Y., He, Y., Xiong, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Zhu, Y. X., Xu, Y., Huang, Y., Li, Y., Zheng, Y., Zhu, Y., Ma, Y., Tang, Y., Zha, Y., Yan, Y., Ren, Z. Z.. Ren, Z., Sha, Z., Fu, Z., Xu, Z., Xie, Z., Zhang, Z., Hao, 7., Ma, Z., Yan, Z.,, Wu, 7.,, Gu, 7.,, 7hu, 7., Liu, Z., Li, Z., Xie, Z., Song, Z., Pan, Z., Huang, Z., Xu, Z., Zhang, Z. and Zhang, Z. Deepseek-rl: Incentivizing reasoning capability in llms via reinforeemmett learning, 2025. URL https:/ www.vv..oggaas.//oo 22500. 12948.
The long-document transformer, 2020. URL https: //aaxavvavv/////////aa org/ wttp:vv.org//bb//000e0 220/ 05150.
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling, 2024, URI. https: /atps://arxiv... org/ abs/ / 2407 007771777..
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 2021a. URL org/abs/4 /abs//11.... 14168.
Delétang, G., Ruoss, A., Grau-Moya, J., Genewein, T. Wenliang, L. K., Catt, E., Cundy, C., Hutter, M., Legg, S., Veness, J., and Ortega, P. A. Neural networks and the chomsky hierarchy, 2023. URL https: //arxiv. org/ abs / 22007. 02098.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert. M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14166, 2021b.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshom, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chemn B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Al- lonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E.,, Smith, E. M., Radenovic, F., Zhang, F., Syn- naeve, G., Lee, G., Anderson, G. L.,, Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I., Misra, I. Evtimov, I., Copet. J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang. J., Yu, J., Bitton, J., Spisak, J., Park, J.. Rocca, J., Johnstun, J., Saxe, J., Jia. J., Alwala, K. V.. Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, K., Chiu, K., Bhalla, K., Rantala-Yean L., van der Maaten, L., Chen, L., Tan, L., Jenkins, L., Martin, L.,, Madaan, L., Malo, L., Blecher, L., Landzaat, L., de Oliveira, L., Muzzi, M., Pasupuleti, M., Singh, M., Paluri, M., Kardas, M., Oldham, M., Rita, M.,
Dao, T. FlastAttention-2:: Faster attention with better paral- lelism and work partitioning. In International Conference on Learning Representations (ICLR). 2024.
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. FlashAt- tention: Fast and memory-efficient exact attention with 10-awareness. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2022.
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q. Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F, Hao, G., Chen, G., Li, G., Zhang. H., Bao, H., Xu, H., Wang, H., Ding, H., Xin, H., Gao, H., Qu, H., Li, H. Guo, J., Li, J., Wang, J., Chen, J., Yuan, J., Qiu, J., Li, J., Cai, J. I.., Ni, J., Liang, J., Chen, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L., Zhang, L.,, Zhao, L., Wang, L., Zhang, L., Xu, L., Xia, L., Zhang, M., Zhang, M.. Tang, M., Li, M., Wang, M., Li, M., Tian, N., Huang, P., Zhang, P., Wang, Q., Chen,
10

---

GSM-o0: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P., He, Q- Dong, Q. Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R. Raileanu, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor. R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabas- appa. S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Her- man, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher. T., Mihaylov, T., Xiao, T., Karn, U. Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Tan, X. E., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z., Chen, Z., Papakipos, Z., Singh, A., Grattafiori, A., Jain, A., Kelsey, A., Shajnfeld,
L., Bell, L.,, Zhang, L., Guo, L.,, Yu, L.,, Moshkovich, L., Wehrstedt, L.., Khabsa, M., Avalani, M., Bhatt, M., Tsimpoukelli, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M. Lathi, M.. Ke- neally, M., Seltzer, M. I.., Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanar, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Laptev, N. P., Dong, N., Zhang, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj. P., Liang, Q. Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Li, R., Hogan, R., Battey, R., Wang, R., Maheswari, R., Howes, R., Rinott, R., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S. Satterfield S., Govindaprasad, S., Gupta, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Kohler, T.. Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Albiero, V., Ionescu, V., Poe- naru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wang, X., Wu, X., Wang, X., Xia, X., Wu, X., Gao, X., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Hao, Y., Qian, Y., He, Y., Rait, Z., De Viiooo, Z., Rosnbrick, Z., Wen, Z., Yang, Z., and Zhao, 7. The llama 3 herd of models, 2024. URI https: /ttp://aaavvve/aa org/ abs/ 24007 21783.
A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Vaughan, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Franco, A., Saraf, A., Chowdhury, A., Gabricl, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C. Wang, C., Kim, C., Zhou, C., Hu, C., Chu, C.-.., Cai, C. Tindal, C., Feichtenhofer, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Wyatt, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D.,, Holland, D., Dowling, E., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, E. Kreuk, F., Tian, F., Ozgenel, F., Caggioni, F., Guzmán, F., Kanayet, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Thattai, G., Herman, Haamiiaaaayaaaann G., Sizov, G., Guangyi, Zhang, Lakshminarayanan, G., Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Damlaj, I., Molybog, I., Tufanov, I., Veliche, I.-E.. Gat, I., Weissman, J., Geboski, J., Kohli, J., Asher, J., Gaya, J.-B., Marcus, J.. Tang, J., Chan, J., Zhen, J., Reizenstein, J.. Teboul, J.. Zhong. J., Jin, J., Yang, J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K. H., Saxena, K., Prasad, K.,
Geva, M., Khashabi, D.,
Segal, E., Khot, T., Roth, D., and Berant, J. Did aristotle use laptop? a question answering benchmark with implicit reasoning strategies, 2021. URL https://arxile org/ / 2101. 02235.
Github. Needle in haystack pressure testing Ilms, 2023. URL https: com/ wtt.oo/gggaaa/////// LLMTest_ Seineett_eeedddellllllattttttttt mmain.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song. D., and Steinhardt, Measuring math- ematical problem solving with the math dataset, 2021. URL htt.sttps//// / www.aaavvvvvoo.... wwwg abs///////// / 221/211.... 03874.
Hsieh, C.PP. Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., and Ginsburg, B. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:240400666 2024a.
Khandelwal, K., Zand, K., Matosich, K., Veeraragha- van, K., Michelena, K., Li, K., Huang, K., Chawla, K., Lakhotia, K., Huang, K., Chen, L., Garg, L., A, L., Silva,
Hsieh, C.P., Sun, S., Kriman, S., Acharya, S., Rekesh,

---

W., Cheng, W., Li, W., Song, X., Su, X., Han, X., Zhang, X., Hou, X., Min, X., Zou, X., Shen, X., Gong, Y., Zhu, Y., Zhou, Y., Zhong, Y., Hu, Y., Fan, Y., Yu, Y., Yang, GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
D., Jia, F,, Zhang, Y., and Ginsburg, B. Ruler: What's the real context size of your long-context language models?, 2024b. URL https: http://axxvvvoo org/ abs/ 2404. 06654.
Natural Language Processing: Industry Track, pp. 881-
893, 2024b.
Liu, H., Zaharia, M., and Abbeel, P. Ring attention with blockwise transformers for near-infinite context, 2023a. URL https: / /awx/aiivvv//////////////////aaa org htwgg///////a / 2310. 01889.
Jiang. A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S.., Casas, D. d. I., Bressand, F., Lengyel. G., Lample, G., Saulnier, I.,, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a.
Liu, N. F., Lin, K., Hewitt J., Paranjape, A., Bevilacqua, M., Petroni, F,, and Liang. P. Lost in the middle: How language models use long contexts, 2023b. URL https: //arxiv. www//as///.....oo.... 22000 03172.
Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi- Yu, J. Yang, Y., Callan, J., and Neubig, G. Active retrieval augmented generation, 2023b. URL htt.s: //rxivv. org/abs, 2305. 06833.
Loughridge, C., Sun, Q., Ahrenbach. S., Cassano, F., Sun, C., Sheng. Y., Mudide, A., Misu, M. R. H., Amin, N., and Tegmark, M. Dafnybench: A benchmark for formal software verification, 2024. URL aittps://arat org/ /bs//b///// 2200 08467.
Kamradt, G. Needle in haystack pressure testing llms, 2023. URL LLMTesttieecdeeenaaassttttekekee manin.
Kuratov, Y. Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., and Burtsev, M. Babilong: Test- ing the limits of Iims with long context reasoning-in- a-haystack, 2024. URL 2406. 10149.
MiniMax, Li, A., Gong, B., Yang. B., Shan, B., Liu, C., Zhu, C., Zhang, C., Guo, C., Chen, D., Li, D., Jiao, E., Li, G., Zhang, G., Sun, H., Dong, H., Zhu, J., Zhuang, J., Song, J., Zhu, J., Han, J., Li, J., Xie, J., Xu, J., Yan, J., Zhang. K., Xiao, K., Kang, K., Han, L., Wang, L., Yu, I., Feng, I.., Zheng, I.,, Chai, I.., Xing, L.,, Ju, M., Chi, M., Zhang, M., Huang, P., Niu, P., Li, P., Zhao, P., Yang. Q., Xu, Q. Wang, Q. Wang. Q., Li, Q. Leng, R., Shi, S., Yu, S., Li. S., Zhu, S., Huang, T., Liang. T.. Sun, W., Sun,
http://wxwvv.rr.eeeo.. abs/
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Lee, J., Chen, A., Dai, Z., Dua, D., Sachan, D. S., Bo- ratko, M., Luan, Y., Arnold, S. M. R., Perot, V., Dalmia, S., Hu, H., Lin, X., Pasupat, P., Amini, A., Cole, R., Riedel, S., Naim, I., Chang, M.-W., and Guu, K. Can long-context language models subsume retrieval, rag, sql, and more?, 2024. URL https: / /aaxxvvvvvvv org/orrg/s.....ooooo... /bs/ 2406 13121.
Y. Li, Y., Huang, Y., Li, Y., Huang, Y., Xu, Y., Mao, Y., Li, Z., Li, Z., Tao, Z., Ying. Z., Cong, Z., Qin, Z., Fan,
Z., Yu, Z.,, Jiang. Z.,, and Wu, Z. Minimax-01: Scaling foundation models with lightning attention, 2025. URL https: /ons://aazzxi ht///axxvvvvrrr/ .rg. hat///ba/225000 2200 08313.
Levy, M., Jacoby, A., and Goldberg, Y. Same task, more tokens: the impact of input length on the reasoning performance of large language models, 2024. URL https://arxii org/ abs//bb///// 220...2222222 14848.
Mirzadeh, I., Alizadeh, K., Shahrokhi. H., Tuzel, O., Bengio, S., and Farajtabar, M. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.0522) 2024.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., tau Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. Retrieval- augmented generation for knowledge-nnennsive nlp tasks, 2021. URL https: //t::i////xvvvi.a 11401.
Qwen, :, Yang. A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang. J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q. Men, R., Lin, R., Li, T.. Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report, 2025. URL https: //arxiv. www.iv..rrg//aa. www.//////////////////..//eeee/e /20os//2412..20 15115.
Li, X., Cao, Y., Ma, Y., and Sun, A. Long context vs. rag for Hms: An evaluation and revisits. arXiv preprint arXiv:2501.00 088000 2024a.
Li, Z., Li, C., Zhang, M., Mei, Q., and Bendersky, M. Re- trieval augmented generation or long-context llms? a comprehensive study and hybrid approach. In Proceed- ings of the 2024 Conference on Empirical Methods in
Shyam, V., Pilault, J., Shepperd, E., Anthony, Q. and Millidge, B. Tree attention: Topology- aware decoding for long-context attention on gpu clusters, 2024. URL https://arxiv www..//rggg////eeeeee 04093.
12

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Sinha, K., Sodhani, S., Dong, J., Pineau, J., and Hamilton, W. L. Clutrr: A diagnostic benchmark for inductive reasoning from text, 2019. URL https:/ //rxiv. org/abs/1908 06177.
Yu, T., Xu, A., and Akkiraju, R. In defense of rag in the of long-context language models, 2024b. URL https: / //aaxiv. wrg. /00.b./2299.....
Zhang. X., Chen, Y., Hu, S., Xu, Z.,, Chen, J., Hao, M. K., Han, X., Thai, Z. L.,, Wang, S., Liu, Z., and Sun, M. ocbench: Extending long context evaluation beyond 100k tokens, 2024. URL https:/ /arxiv.vvvo///o//aaaaaaaa wrw. hagg/bb/// 2402.13718.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than seal- ing model parameters, 2024. URL https: /atps://araas //arxiv...vv org/ atg/abs/// abs/2408 2408. 03314.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang. S., ct al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens context. arXiv preprint arXiv:2403.05530, 2024a.
Team, J., Lenz, B.. Arazi, A., Bergman, A., Manevich, A., Peleg, B., Aviram, B., Almagor, C., Fridman, C., Padnos, D., et al. Jamba-1.5: Hybrid transformer-mamba models at scale. arXiv preprint arXiv:2408. 12570, 2024b.
Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. Musique: Multihop questions via single-hop ques- tion composition, 2022. URL abs/2108 00573.
org/
Vodrahalli, K., Ontanon, S., Tripuraneni, N., Xu, K., Jain, S., Shivanna, R., Hui, J., Dikkala, N., Kazemi, M., Fatemi, B., Anil, R., Dyer, E., Shakeri, S., Vij, R., Mehta, H., Ramasesh, V., Le, Q., Chi, E., Lu, Y., Fi- rat, O., Lazaridou, A., Lespiau, J.-B., Attaluri, N., and Olszewska, K. Michelangelo: Long context evaluations beyond haystacks via latent structure queries, 2024. URL https: / aaaiiiii.. org/ abs 2.09.100 12640.
Wiles, A. Modular elliptic curves and fermat's last theorem. Annals of mathematics, 141(3):443-55, 1995.
Yang, Z. Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question an- swering, 2018. URL https: //axxv////iivv///a org/ /bs/ 1809.09600.
Ye, T. Xu, Z., Li, Y., and Allen-Zhu, Z. Physics of language models: Part 2.1, grade-school math and the hidden rea- soning process, 2024a. URL https:////// /aaxxiiiv.../////// org/ abs/2407.20311 20311.
Ye, T. Xu, Z., Li, Y., and Allen-Zhu Z. Physies of language models: Part 2.2, how to learn from mistakes grade- school math problems, 2024b. URL https: arrxiv. hog//bs//en 2408 16293.
Yu, T., Xu, A., and Akkiraju, R. In defense of rag in the era of long-context language models. arXiv preprint 2xXiv2449.00166 2024a.
13

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Appendix Table of Contents
present the following table of contents to better traverse the Appendix.
Related Work
Long-context Language Models
Long context benchmarks and tasks
Limitation of Existing Reasoning Tasks
Synthesized Datasets for Long-context
Benchmark Limitation
Detailed Experiment Setup
RAG Experiment Setup
Detailed Experiment Metries
Repeated Sampling Experiment Setup
Full Result of RAG Experiments
Illustrative Problems
Forward and Reverse Problems Breakdown
Ablation Study of Task Templates
Ablation Study of Graph Structure
Detailed Model Comparisons
Cost Estimation
Impact of Problem Structure: Forward vs. Reverse Reasoning
Long-Context Degradation of Models
Common Error Analysis
A. Related Work
A.1. Long-context Language Models
Various works related to the Long-context Language Model have been proposed. Flash attention(Dao et al., 2022), Flash attention2(Dao, 2024), Ring attention(Liu et al., 2023a), and Tree attention(Shyam et al., 2024) significantly reduced the memory footprint and communication overhead processing long context in engineering level across multiple nodes. Architectural level innovations such as sparse attentions represented by sliding window attentionnBellagy al., 2020). also widely used reduce the overhead caused by the increasing sequence length. New training strategies, such as gradually extending the training context length in the final stages of pretraining have been applied to support a long context windyw(Dubey et al., 2024).
A.2. Long context benchmarks and tasks
There have been quite few works benchmarking long-cor long-context language models. Existing comprehensive benchmarks like oxbench(Zhang et al., 2024) cover realistic tasks including document QA, summary, and synthetic tasks including information retrieval, expression calculation, extending the context length in the benchmark to over 200k tokens. oobench(Zhang et al., 2024) does have mathematical reasoning tasks, however the most relevant math.calc part seems to be too difficult for SOTA models to work out. Synthetic tasks often offer more control and are less affected by parametric knowledge in comparison with realistic tasks. One comprehensive synthetic benchmark RULER(Hsiee al., 2024a), synthetic benchmark with
14

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
tasks including retrieval, variable tracking and so on, offering some controls over context length and task complexity, Experiments with various complexities were done, but it does not provide a quantitative analysis of complexity and context length on the correctness of the task, let alone isolate two separate patterns of performance decay. Other benchmarks usually focus on simple retrieval(Github, 2023; Liu et al., 2023b), fact reasoning(Kurator et al., 2024), the impact of long context on natural language reasoning(L_evy et al., 2024) and other real-world knowledge involved tasks.
A.3. Limitation of Existing Reasoning Tasks
Popular reasoning benchmarks are loose collections of buman-made problems that naturally suffer from the following limitations. Firstly, the difficulty of problems within the same benchmark varies widely. We analyzed all 1.3K test problems in the GSM8K dataset, we plot the histogram in the number of operations in Figure 4(a). The problem varied from 2 to over 12 following a skewed bell shape curve. This lack of fine-grained control makes it challenging to systematically evaluate models across incremental difficulty levels. Also, notice that the total number of problems is less than 10 for op 2 9, too little for stable evaluation. The lack of problem quantity on human-curated datasets eliminates the possibility of filtering out problems of each fine-graines difficulty level. Secondly, there is significant difficulty gap between the benchmarks: GSM- 8K focuses on middle school problems, MATH (Hendrycks et al., 2021) and AIME targets prospective university students, and Frontier Math challenges top-tier math graduate students. It is difficult to quantitatively determine the difference in problem difficulty between GSM-8K problems with MATH problems since MATH uses operations such as taking power or roots that are absent in GSM-8K. Besides, there exists a rich set of common-sense reasoning benchmarks( Saang et al., 2018; Trivedi et al., 2022; Geva et al., 2021). However, the number of hops are usually limited to two to four, which although natural and requires LLM commonsense, is significantly limited in complexity. Similarly, it is not possible to determine the difference in complexity from MATH to Frontier Math. It is difficult to quantitatively model LLMs' performance degradation with the continuously increasing difficulty of the problem. Third, most of the existing problems have very short input prompts. On average, GSM-8K test set problems have a length of 59.96 tokens, while MATH test set problems have 67.37 tokens when using Llama 3.1 tokenizer. We have seen from 2 that the addition of irrelevant noise cannot meaningfully evaluate the ability to reason in a long context of LLMs.
A.4. Synthesized Datasets for long-context
Synthesized tasks are simple to build and absolutely deterministic, data contamination safe, but highly effective to evaluate certain aspects of LLM performance. Its use in long-context benchmarks is profound. beedle-in-the-haystace (Kamradt, 2023), pioneering long-context task, becomes the go-to task for evaluating LLM long-context retrieval ability. On the other hand, LLM reasoning benchmarks also see recent efforts in synthesized tasks. (Sinha et al., 2019) proposes to evaluate language models through description of kinship of many people, which can be naturally formulated in graphs. Although the problem generator proposed require human annotation for ground-truth and impose no absolute guarantee in correctness of the problems, preventing scaling up of the method. (Mirzadeh et al., 2024) recently proposes to use build synthesized dataset upon GSM8K (Cobbe et al., 2021b) to study the robustness of LLM reasoning. Part of our work draws a strong inspiration from a series of works ((Ye et al., 2024a), (Ye et al., 2024b)) which systematically studies the intricacies of decoder transformers in solving grade-school level problems. Following their footsteps, we carefully redesign the process of generating the problems so current LLMs can solve without training, and together with thoughtful steps in noise addition, we effectively construct effective reasoning benchmarks for the long-context community. Concurrently, (Vodraballi et al., 2024) also introduce meaningful noise into short essential problem sentences to probe long-context ability of LLM. Differently, we also focus on scaling reasoning complexity (thops), which help me present significantly different insights from their study into understanding both scaling up meaningful context length and reasoning complexity continuously.
B. Benchmark Limitation
Despite our careful construction and attention LLM understandability, there are still several important limitations the existing benchmark GSM-20. First, our current benchmark only targets operations contained in the grades school math, so only +, - x, + are included. We are aware that Math (Hendrycks et al., 2021) includes more complex operators, which we are looking at incorporating into the next version of the benchmark. The key limitation it brings to our benchmark is shown in Figure 2(a), to generate complex problems with high reasoning complexity, our generator needs to generate problems that are much longer than hard mathematic benchmarks, despite our limitless quantity and complexity scalability. Second, another key limitation of synthetic benchmarks is the lack of diversity in natural language. Natural Language reasoning tasks usually contain multiple ways to assign a variable value or to describe a relationship between multiple variables, which
15

---

GSM-co: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
extremely difficult program in software without the LLMs in the loop. We deliberately aim to keep LLMs out of the loop for better scalability and counter the diversity by incorporating more than one template into the problem generation. Third, we implement multiple checks to ensure all of the essential steps in the core graph have positive values and have number range of less than four figures, but for the current noise generation, we don't implement these checks, since they are redundant to the problems. However, we do notice that when processing through the noise, the L.LMs sometimes are confused by the irregular noise variable values. In later version of the benchmark, we will address this issue and make sure the entire graph consistent.
C. Detailed Experiment Setup
C.1. RAG Experiment Setup
The RAG system contains two components, the retriever and the decoder. For the retriever, we use all-mpnet-v2-base For the decoder, we use Llama-3. input context as the data store (no external source). We segment the input text into chunks (roughly a sentence) using NLTK package. The context retrieval budget for all problems is 2048. We employ two different RAG methods: passive and active RAGs. Passive RAG calls the retriever once before the generation of the decoder. The retriever computes the semantic similarity or distance between each chunk of context and the query sentence. These chunks in context are then ranked from closest (most semantic similar) to furthest (least semantic similar), and depending the retrieved context, top-k chunks retrieved. For our study, we used the L2 distance between context chunk embeddings and query embeddings. The decoder then takes the retrieved chunks as input and then outputs its response to the query.
We follow the convention of context-level RAG method. We solely use the
We use two types of RAG systems: Passive RAG that only calls the retriever once at the beginning to retrieve relevant context or Interactive RAG (Jiang et al., 2023b) in which the decoder decides when to retrieve, how many retrievals are needed, and generate query for each retrieval. For the latter one, we restrict the decoder generation with only the latest retrieval content and its past generation.
On the other hand, active RAG shows strong performance (Jiang et al., 2023b) especially for common sense reasoning tasks. In addition to the steps in passive RAGs, the decoder is allowed to initiate additional calls to the retriever to retrieve more context by generating new queries. We follow the state-of-the-art active RAG method FLARE (Jiang el al., 2023b), which allows for 10 rounds of query, but restricts the LLM to only see its current round of retrieved context and its past rounds of generation to generate its full response.
C.2. Detailed Experiment Metrics
The Area Under Curve (AUC) metric, conceptually illustrated in Figure 10, quantifies model performance by integrating accuracy over the "Number of Ops." First, accuracy measurements (14) are collected at corresponding "Number of Ops" values (I,). These (z,, y:) pairs are then sorted based on the x; values. The AUC is subsequently computed using the trapezoidal rule on these sorted points. For N data points (Ir, yo),..., (1N-1;9N-)) sorted by x, the raw area is given by:
N-1
Ui + +i Vi-1 (0x- x2-1)
AUCraw raw
This sum approximates the area under the curve formed by connecting consecutive (Accuracy, Number of Ops) points. The final reported AUC value is then scaled:
AUC = AUC x 100
(1)
A higher AUC indicates superior overall accuracy maintained across the evaluated range of "Number of Ops." The "Acc Threshold" in the figure serves as visual performance benchmark. While Figure 10 depicts rectangular "boxes" for illustrative simplicity, the actual numerical integration employs the more precise trapezoidal method described.
C.3. Repeated Sampling Experiment Setup
C.3.1. PROCEDURE
1. Oversampling Phase
Generate 256 samples per task with temperature T 1.0
16

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Measurement
Acc Threshold
Sum of of the Boxes are AUC
Accuracy
Number Ops
Figure 10. Area Under Curve Metrics is Used to Compare between LLM Performance.
Use fixed random seeds for reproducibility
2. Accuracy Calculation
Compute per-task empirical accuracy:
# Correct Samples
Pusk
(2)
256
Estimate accuracy for N samples:
AcCrask 1 - (1 Puask N
(3)
3. Aggregation
Average results across 80 tasks:
Final Accuracy
AcCask
(4)
80
C.3.2. RATIONALE
Oversampling: 256 samples reduces variance estimating Ptak compared to using 128 samples directly
Probability Formula: Models cumulative success probability:
P correct in & trials) = - p)
p)*
Task Count: 80 tasks per op provide stable statistics while remaining computationally feasible
17

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
D. Full Result of RAG experiments
Here, we present the full result of the RAG experiment for further analysis. We retrieve 2048 tokens for each RAG retrieval.
D.1. RULER
Models
sl
s2
83
mk1
mk2
mk3
mv
mq
Context Length
Llama 3.1 70B Instruct
100
100
100
100
100
100
100
100
8k
OnePass RAG
100
100
100
100
100
96
98.5
100
8k
Interactive RAG
100
100
100
100
100
99
99
Llama 3.1 70B Instruct
100
100
100
100
98
100
98
100
32k
OnePass RAG
100
100
100
98
100
72
99.5
97.5
32k
Interactive RAG
100
100
100
98
96
96
98.5
98.5
32k
Llama 3.1 70B Instruct
100
100
100
98
96
100
89
98
64k
OnePass RAG
100
100
100
100
100
56
95.5
100
64k
Interactive RAG
100
100
100
100
96
98
99
100
64k
Table 3: RAG vs Model (RULER NIAH)
Models
vt
cwe
fwe
qal
qa2
Context Length
Llama 3.1 70B Instruct
100
100
96.67
84
74
8k
OnePass RAG
82.4
14.8
97.33
86
86
8k
Interactive RAG
98.4
31.2
79.33
80
8k
Llama 3.1 70B Instruct
100
95.2
97.33
80
66
32k
OnePass RAG
86
5.2
92
84
74
32k
Interactive RAG
98
7.6
80
78
64
32k
Llama 3.1 70B Instruct
100
6.2
95.33
72
62
64k
OnePass RAG
78.4
1.2
88.67
82
74
64k
Interactive RAG
98.8
2.6
72
78
56
64k
Table 4: RAG vs Model (RULER other subsets)
Abbreviations: s1-3 = niah_single.1-3 mk1-3 = niah wwwpmmmmlllllle13 113, mv = niah multivalue, mq = niah muiliquu
D.2. LongBench V2
Tasks
Overall
Easy
Hard
Short
Long
Llama 3.1 70B Instruct
30
33.3
28.1
44.7
OnePassRAG
25
33.3
20.3
23.7
25.8
InteractiveRAG
33
36.1
31.2
34.2
32.3
Table RAG Model (LongBench V2)
D.3. LongBench
Tasks
pausage.count count 36.0,36.0,32.0 58.87,71.22,76.44
hotpot-qa
samsum
Llama 3.1 70B Instruct
28.88,35.55,41.48
OncPassRAG
0.0,0.0,0.0 27.0,14.0,6.0
65.04.61.59.63.05 61.86.51.0.55.94
31.63,23.28.26.88
InteractiveRAG
24.83,20.21,23.88
Table 6: RAG vs Model (LongBench). Part
18

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Tasks
multi-news
multifieldqa.co en
gov rev_report
Llama 3.1 70B Instruct
27.71.24.11,23.17 26.85.22.72.20..99 51.69,48.66.55.88
57.31,51.83,64.88
34.94,34.7,31.82
OnePassRAG
32.6,30.67.27.32
InteractiveRAG
47.71,42.98.58.45 14.64,19.99,18887
29.91,27.02.25.34
Table RAG vs Model (LongBench) Part 2
Tasks
qasper
passage retiivvellriev
2wikimqa
Llama 3.1 70B Instruct
50.3,46.5,25.89 45.73,43.5.35.3
100.0,100.0,1001
74.93,64.37,5966
OnePassRAG
72.0.72.0,79.0
67.97,59.64,48.4
InteractiveRAG
43.8,37.9,32.88 84
91.0,90.0,85.33
46.04,43.46,3888
Table 8: RAG vs Model (LongBench) Part
Tasks
triviaqa
trec 48.0,12.0,122,
lcc
repobench-p
Llama 3.1 70B Instruct
82.0,93.6,94.0 92.13,89.46, 47.0,56.0,53.0
50. 50.14,55.0,50044
29.98,27.82,26.
OnePassRAG
19.92,14.5,18.36 56.0,57.0,52.0
34.76,33.62,28.0
InteractiveRAG
88.11,92.32.91.5
24.26,23.26,22.57
14.97.17.15.16.56
Table 9: RAG vs Model (LongBench)- 4art
Comments: The 3 data separated by commas are subsets of 0-4k, 4-8k,8k+ respectively
D.4. LOFT
Tasks
ArguAna
FEVER
FIQA
MS MARCO
NQ
Quora
SciFaat
Llama 3.1 70B Instruct
0.06
0.78
0.37
0.67
0.84
0.62
0.59
OnePassRAG
0.64
0.88
0.45
0.77
0.86
0.62
0.64
InteractiveRAG
0.42
0.73
0.53
0.69
0.76
0.83
0.87
Table 10: RAG Model (LOOF)) Part
Tasks
Touché-2020
HotPotQA
MuSiQue
QAMPARI
QUEST
Llama 3.1 70B Instruct
0.4411
0.37
0.2
0,024
0.07166
OnePassRAG
0.2529
0.455
0.2383
0.1559
0.1899
InteractivRAAG
0.79
0.29
0.13
0.1539
0.2983
Table 11: RAG vs Model (LOFT) Part 2
Comments: Due to LOFT's limited prompt availability (no official releases for 32k/IM contexts), we conducted experiments solely on 128k context lengths. We observed that LOFT's document ID) retrieval tasks inherently require document identifiers that aren't 't captured in standard RAG-retrieved chunks. To address this, we implemented lightweight modification: appending document ID tags to each context chunk during retrieval. This allows the RAG system to infer the correct document when retrieving relevant content, resolving the task-specific limitation without altering core RAG functionality.
19

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
E. Illustrative Problems
This section presents one representative problem from each subset (Symbolic, Medium, and Hard) defined in the appendix. These examples illustrate the variations within the benchmark. see Table 12
Table 12. Illustrative Problems from Each Subset
Feature
Symbolic
Medium
Hard
Problem
Symbolic (op=5): <context>> nassign V705804 V437110 assign V986916 V705804.
assign V873548 = 6. assign V684196 = V873548. assign V437110 = \n </context> nn/nTh context contains relationships between variables. These relationships are independent mathematical equations that are all satisfied simultaneously. \n Using only these relationships, determine which variables (if any) from which values can be derived are equal to 7. AnSSow your step-by-step reasoning and calculations, and then conclude your final answer in a sentence.
Answer: V705804, V986916.
Medium (op=5):
The number of adult owl in Bundle Ranch equals 2 times the number of adult eagle in Bundle Ranch. The number of adult eagle in Hamilton Farm equals the difference between the total number of adult animals in Bundle Ranch and the number of adult eagle in Bundle Ranch. The number of adult owl in Hamilton Farm equals 4 times the number of adult owl in Bundle Ranch. The number of adult eagle in Bundle Ranch equals Question: What is the total number of adult animals in Bundle Ranch? Answer: 9.
Problem:
Hard (op=5); The average number of newborn children per adult blue jay in Bundle Ranch equals The number of adult parrot in Bundle Ranch equals 2. The number of adult blue jay in Bundle Ranch equals times the average number of newborn children per adult blue jay in Bundle Ranch. The number of adult eagle in Bundle Ranch equals 2 times the average number of newborn children per adult blue jay in Bundle Ranch. The number of adult parrot in South 7o0 equals 4 times the sum of the average number of newborn children per adult eagle in Hamilton Farm, the number of adult eagle in Hamilton Farm, and the average number of newborn children per adult eagle in Hamilton Farm. The average number of newborn children per adult eagle in Hamilton Farm equals the number of adult cagle in Bundle Ranch. The number of adult eagle in Hamilton Farm equals The average number of newborn children per adult parrot in Bundle Ranch equals the total number of adult animals in Hamilton Farm. The number of adult eagle in South 7oo equals 1. The average number of newborn children per adult parrot in South Zoo equals the average number of newborn children per adult parrot in Bundle Ranch. The average number of newborn children per adult eagle in Bundle Ranch equals 3 plus the average number of newborn children per adult parrot in Bundle Ranch. The average number of newborn children per adult cagle in South Zoo equals the sum of the number of adult blue jay in Bundle Ranch, the average number of newborn children per adult blue jay in Bundle Ranch, the average number of newborn children per adult parrot in Bundle Ranch, and the number of adult parrot in Bundle Ranch. Question: What is the average number of newborn children per adult eagle in Bundle Ranch? Answer: 6.
20

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
F. Forward and Reverse Problems Breakdown
Models
Forward Problem
Reverse Problem
Forward AUC Reverse AUC
Llama3. 1a0taa.1-70O-nnttteee
62000000000
1283.750000
816.875000
GPT-44-mini
1529. 725400
1267.5 2267057
262,1 146400
L.5aa-ee
390.380000
624.980000
6234.000000
GPT-4o
3073.997375
1952.816875
1121.180500
Mistral-Large
3468.2344 2188.234000
2431.732453
1036.501650
Llama3. 1-8B-Instruct
1030,000000
563. 125000
466.877 466.875000
Claude-3.5-Somee
3653.8 830050
3158.657850
495. 172200
Qwen2.5-72B-Ins Instrructt
2889.375000
2141.25000
748. 125000
Qwen2.5-7B-Inse
995.625000
833.125000
162.500000
ol-mini
6517.510550
5592.307 100
925. 203450
Gemini-1.55 1.5-aii-.5-
1889.375000
1153,750000
735. 625000
Claude-3.5 H.5-Hakuu
1234.622 1234.620000
873.100000
361.520000
Llama-3. InsttutInttruct
100.000000
981 981,250000
800. 50000
DeepSeek-V V3
4613. 125000
3713. 125000
900,000000
Gemini- 1.5-Pro-002
4204.564075
3160.574950
1043.9891 125
DeepSeek-R2 R1
9764. 950000
9750.950000
14. 4.,000000
MiniMax Text-01
2148.07 1300
415650
408.655650
QwQ-32B-Previce
3530.000000
2846.250000
683.750000
Table 13: Medium Difference in AUC in Forward Problems and Reverse
Problems
Models
Forward Problem
Reverse Problem
Forward AUC
Reverse AUC
Claude-3.5- Haiku
819.240000
776,900000
42.340000
Llama3. 1-r...1--0-Intttruee
1314.375000
1098.75000 750000
215.625000
1ommini-.55-lash-ee
1341.250000
1219.375000
121.875000
MiniMax Text-O1
1360.555000
1034.625000
325.9300K
DeepSeek-RI
8444.500000
8756.950000
-312.450000
ol-mini
3831.38100 1000
6645.444420
185, 185,906800
Gemini- -5-Prr-002
2255.732025
2444.270375
-188,538350
DeepSeek-N V3
2725.085000
2109.560000
615.525000
Qwen2.5-7B-Instee IIstruct
625.625000
630.625000
-5,000000
GPT-4o
1592.280000
13100000000
280. 720000
Llama3. --8-Inntt
759.375000
460.625000
298. 750000
Qwen2.5-72B. Bwen2.5-72B-Instruct Instruct
21960875000
08000000000
301.875000
Claude-3.5-Sonnet
2242. 1242.309950
1999.99 .981 100
242.311855
GPT-4o-mini
858.400000
873.310000
-14.910000
QwQ-32BB --eeeiee
1878.750000
1855.6255 625000
23,125000
Mistral-Large
2570.940500
2018. 469000
552.471500
Ilama-3.. 1405B-Instrucc
1215.000000
753.750000
471, 471,250000
Jamba- Large
274.980000
699.990000
-425.010000
Table Hard Difference AUC in Forward Problems and Reverse
Problems
21

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Despaeee/33 on Zaro
Lame-33 1-4258- ratrust or Zaro
MeeratHange-211 Masa Hard
sepo
witt Reverse
Proeet only
100
Femerl in
ualy
30
04
02-
Mi Miriundliesidd un Zaro Noias Hediam
Qwean2 5-728-bbdrutt un Zaru hoies Radiam
Seporation
Separation
Bevare of
30
Figure 11. Comparison between forward and reverse
G. Ablation Study of Task Templates
We have three different real-world templates ready. We show that they offer consistent scores with Llama-3.1 3.1-BBBBBBBBB----- 1-8B-Inntruct with slight variables in specific operations. We also show three problem examples.
22

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Template: Zootopla"
Comperativve Performence Different Mediums
Problem: number adult racoon Zoo equais plus total
Zootopia Medium
adult animalls in Mayer Aquarium. The number of adult fox Meyer Aquarium equells 2. the
Teachers School Medium
Question:
enimals
Movie Festivals Medium
Solution: Define adult Mayer Aquarium as sot- Define total number edult
animals in Mayer Aquarium 60 L=t= Dehin adult mecoon South Zoo ht = Define
n
animals
Y
3. Answar:
Template: " Tesccees in School"
Problem: number ongiona medical schoolin Brightford alementary
number Hawkesbury equaia plus total schoola
Gb
Brightoord.
Question:
schools
Solution: Define regional medical echool in Brightford as Define total
number of schopls Brightoord as
Define elementary school in Define number schools
Hewkesbury
Hewkesbury as tz so
Answer: 2.
Template: "Movie Festival Awards"
Problem: The number solemn period drama in Festival de Ciaiment equals plus Saint-Rivage.
C2
road
Festival de Saint- Rivage aquals
Question: What the total number of movies Festival de Clairmont?
Solution: Define
Saint-Aivage
number movies in Festival de Saint-Rivage as x s0 - Define solemn period
Festival Claimont
Define total
Claimont G;
Figure 12, Comparison between different task templates
H. Ablation Study of Graph Structure
To analyze how graph structure impacts model performance, we ran Qwen2.5-7B-Inseee on 40,000 samples from our Symbolic subset with op=9. The Symbolic subset was chosen as it allows us to control for various factors and isolate the impact of graph structure. In this analysis, we focus on the relationship between graph depth and model accuracy. We define the depth of node its distance from the root node. The depth of graph (referred as "Depth") is determined by the maximum depth of its nodes, while the mean depth represents the average depth across all nodes in the graph. We categorized samples based on their maximum graph depth, segmenting the data into five major categories: Depth=2, Depth=3, Depth=4, Depth=5, and Depth=6. Categories with insufficient data (e.g., fewer than pre-defined threshold of samples) were excluded to maintain statistical significance. The results, presented below, confirm an inverse relationship: accuracy generally decreases as the maximum graph depth increases, supporting the hypothesis that deeper graphs are more challenging for the model.
Depth
2
Accuracy 0.438
0.386
0.347
0.317
0.257
Further analysis investigates the relationship between mean depth and accuracy. As shown in Figure 13, there is a clear trend: accuracy tends to decrease as mean depth increases. This inverse relationship suggests that increasing graph complexity, as measured by mean depth, can negatively impact model performance. Notably, when comparing graph groups with similar mean depths, the maximum depth (Depth category) appears to have less significant impact accuracy. This finding implies that it is the overall complexity represented by the mean depth, rather than the specific maximum depth of the longest path, that primarily influences performance.
23

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Accuncty Eopa by Dopth
10
Des (mw4853)
sap2tmal (8.01)
Depitrrh Ine33100 1Mp7) 11-44:0
Maun
Figure 13. Trend of accuracy versus mean depth, categorized by maximum graph depth (Depth). Error bars indicate 95% confidence intervals.
I. Detailed Model Comparisons
Here we present and highlight experiment results, potentially complementing those Table and following the discussion of Section We focus on three main aspects of inter-model comparisons.
First, reasoning-enhancee models demonstrate significant performance advantage their non-reasoning counterparts, with identical parameter counts and model architectures. The following table illustrates this comparison across different operational complexities ('op'):
op
DeepSeek R1
DeepSeek V3
Qwen QWQ
Qwen 2.5 32B Instruct (non-reasoning)
32B Preview
10
0.9453
0.7850
0.6400
0.6100
20
0.9375
0.6000
0,5950
0.3600
30
0,9453
0.3550
0,2550
0.1200
40
0.9231
0.1625
0.1250
0.0437
AUC
8573.800
2407.855
1846.187
1399.805
Second, within the same model family, larger models generally exhibit higher performance than smaller models. This trend evident in the comparisons below for Qwen and Llama families:
op
Qwen 2.5
Qwen 2.5
Llama 3.1
Llama 3.1 8B Instruct
72B Instruct
7B Instruct
70B Instruct
0.9500
0.5850
0.7300
0.5200
10
0.8150
0.2850
0.6400
0.3950
15
0.7550
0.1800
0.4900
0.1900
20
0.6400
0.0500
0.3250
0.0700
25
0.4850
0,0250
0,2625
0,0600
30
0.3650
0.0000
0.0375
0.0200
AUC
2016.375
618.500
1205.250
606.500
Third, the results also indicate a performance gap for models with hybrid architectures when compared to those utilizing a standard Transformer architecture. The following table details these findings, comparing Area Under Curve (AUC) scores:
24

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Models
Architecture
Date of release
Parameter Size
AUC
Qwen-2.5-32B Innttuuttt
Transformer
09/17/2024
32B
1405.055
MiniMax-Texx
Hybrid (Linear Attention)
01/14/2025
456B MoE (45.9B per token)
1178.510
Jamba-1.5 1-5--ar
Hybrid (SSM)
08/22/2024
398B MoE (98B per token)
466.400
25

---

GSM-oo:
How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
J. Cost Estimation
evaluate at least 100 examples for each operational complexity ('op') setting in GSM-Infinite across our three distinct subsets: Symbolic, Medium, and Hard. To illustrate our evaluation scope. consider 1amma-..-770-Imm capture the full performance spectrum, from 'op = 2' (lowest reasoning complexity) until the model's accuracy drops below 0.05. This typically involves evaluating up to 40 different "op" settings for each subset; specifically, the Symbolic subset often requires up to 55 "op" settings, the Medium subset up 45, and the Hard subset up to 30.
We aim to
Our sampling strategy for "op" values varies:
Zero-context evaluations: For most models, evaluate every 'op' from to 30. Beyond 'op 30", where accuracy for models around 70B parameters typically decays significantly, we evaluate one "op" setting per five consecutive 'op' values, adjusting this based on observed model performance.
Longer context evaluations: To manage inference time and cost, we evaluate every second 'op" setting within the first 30 'op' values.
Across the subsets, this translates to approximately 35 evaluation runs (distinct 'op" settings) for the Symbolic subset, 33 runs for Medium, and 30 runs for Hard. For all context length settings, we allow up to 4000 tokens for generation, though for many API-based reasoning models, this output limit cannot be explicitly specified. For zero-context evaluations, the input length is typically upper-bounded at 1000 tokens per example.
Under worst-case estimate for zero-context evaluations, the total number of examples would be (35+333++00) x 100 = 9800 examples. Based on this, we estimate approximately 9.8 million input tokens and 39.2 million output tokens for the zero- context setting. Using representative API costs for 70B model (e.g., $0.12 12 per million input tokens and $0.30 per million output tokens, noting that local deployment can be virtually free direct API costs), we summarize the estimated costs for evaluating 3.ae--..---------s-- across different context lengths in Table 15.
Table 15. Estimated Evaluation Cost for Llama-3, 3.1-
Context Length
Input Tokens (M)
Output Tokens (M)
Input Cost ($)
Output Cost ($)
Total Cost ($)
Zero Context
9.8
39.2
1.18
11.76
12.94
8K
21.6
10.8
2.59
3,24
5.83
16K
38.4
9.6
4.61
2.88
7.49
32K
80.0
10.0
9.60
3.00
12.60
Total
38.86
Our experiments consistently show that larger reasoning models are generally more costly to evaluate, while smaller or less complex models are less expensive.
Furthermore, we investigated the effectiveness of using strides in 'op" selection and reducing the number of samples per 'op" to potentially lower evaluation costs without significantly impacting the Area Under Curve (AUC) metric. For this study, we used the zero-context Hard subset and Qwwen-25532B-IInttee performance saturate. We evaluated scenarios with different stride values, shown in Table 16. The results indicate that strides up to 4 (evaluating every 5th 'op' setting) introduce only a minor effect on the overall AUC.
which required 42 runs (from 'op 2" to 'op = 42') for its
We typically use 200 samples per 'op" for our standard measurements, To explore further cost reduction, we simulated scenarios with 100 and 50 samples per 'op" using probability bootstrapping. As shown in Table 17, using fewer samples can sometimes yield similar AUC values, offering another avenue for reducing evaluation costs.
26

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Table 16. Rffectt of Stride in 'op' Selection AUC (Qwenn-25-32B-mmstmuc,, Zero-Context Hard Subset).
Stride Configuration
AUC
Difference (%)
Stride 0 (cvaluating all ops)
1405.1
Stride 0 (ops 2-30), Stride = (ops >30)
1401.6
-0.25
Stride (evaluating every 2nd op)
1413.1
0.57
Stride 2 (evaluating every 3rd op)
1382.3
-1.62
Stride 3 (evaluating every 4th op)
1404.99
-0.01
Stride 4 (evaluating every 5th op)
1369.5
-2.53
Table 17. Effect of Sample Size per 'op' on AUC (Simulated using Bootstrapping).
Samples Taken per 'op'
AUC
Difference ( (%)
200
1405.1
100
1419.0
0.99
50
1387.0
-1.29
K. Impact of Problem Structure: Forward vs. Reverse Reasoning
The distinction between forward and reverse reasoning problems in our dataset does not primarily concern the explicit arithmetic operations used (e.g., plus, minus, multiply, divide), as these appear in all subsets (referencing Section 4.3). Instead, the core difference lies in the implicit operations and relationships embedded within the problem structure. As illustrated in Figure 6, implicit addition can represent class-instance relationships, while implicit multiplication can represent hierarchical instance dependencies.
Forward problems exclusively contain these two types of implicit operations (constructive dependencies). Consequently, their computation graphs typically show flow where more concrete, detailed instances are computed first, and these are then aggregated to determine more general or summary variables.
In contrast, reverse problems involve implicit "minusing" or "divisions." In these scenarios, values for more abstract variables are often provided, and the language model must deduce unknown instance variables. Consider the following simplified examples:
Forward Problem Example: The number of pigs in Jefferson Park is 2. The number of sheeps in Jefferson Park is 4. Assume animals not mentioned for this location are not present. What is the total number of animals in Jefferson Park? Solution Path: Number of pigs (instance) + Number of sheeps (instance) > Total animals (abstract).
Reverse Problem Example: The number of pigs in Jefferson Park is 2. Sheep are present in Jefferson Park in a non-zero quantity. Assume animals not mentioned for this location are not present. The total number of animals in Jefferson Park is 6. How many sheep are in Jefferson Park? Solution Path: Total animals (abstract). Number of pigs (instance) - Number of sheeps (instance).
As detailed in Appendix F, most evaluated LLMs, with some exceptions, achieve better scores forward problems than on reverse ones, given the same level of reasoning difficulty (i.e., the same number of operations, 'ops'). We hypothesize that this disparity stems from the nature of human- generated text; humans naturally tend to write and explain concepts in the constructive logical ordering characteristic of forward problems. This could lead to a relative scarcity of training data exemplifying reverse logic compared to forward logic.
To investigate whether targeted training can mitigate this gap, we conducted experiments with Owen-2.5-500M-IImmn Instruct.. We prepared a dataset of 1.3 million diverse problems, ranging from 'op=2' 'op=30', generated using our proposed generator (with templates differing from the test set). This training set was balanced, with an equal number of forward and reverse problems (650,000 each).
Our findings indicate that the off-the-shed 500M model initially demonstrates better forward reasoning ability within its effective performance range (accuracy 2 0.05). Beyond this range, its responses tend towards guessing. Interestingly, for reverse problems, due to a potentially smaller search space for the queried variable, the model might exhibit a slightly higher
27

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
probability of correctly guessing in this guessing regime, leading deceptive slight increase in reverse problem accuracy higher "ops'.
After fine-tuning on the 1.3M example dataset, the 500M model showed significant performance improvements on GSM- Infinite. Notably, its performance reverse problems became slightly higher than forward problems. suggesting that providing sufficient relevant training data can substantially enhance the model's reasoning capabilities and largely erase the observed forward-reverse performance gap. The results are summarized in Table 18.
Table Performance Comparison 2.cen22.-5000. -nntruuttt Before and After Fine- tunint on Forward and Reverse Problems.
Evaluations
Qwen2.5 500M Instruct
Qwen2.5 500M Instruct
(op)
Finetuned with 1.3M Examples
Forward
Reverse
Score
Forward
Reverse
Score
0,2128
0.1385
0.1612
0,3032
0.5846
0.5009
0,1093
0.0743
0.0855
0.6721
0.4505
0.5915
0,0302
0.0491
0.0405
0.6604
0.7791
0.7547
10
0.0084
0.0390
0.0311
0.6681
0.6780
0.7076
15
0.0000
0.0690
0.0346
0.5923
0.6767
0.6451
20
0.0000
0.0698
0.0332
0.5207
0.5488
0.5206
25
0.0000
0.0533
0.0317
0.3413
0.3413
0.3506
28

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
L. Long- Tog-Cootext Degradation of Models
In this section, we provide the accuracy decay curves of all LLMs tested across zero-context, 8K, 16K and 32K for further analysis. We selected the first 30 reasoning steps to truncate the data for comparison purposes.
Drostiot-50 Sembets)
Scnte2 2:hhe000 sPed ant
Micreliaage 111 Syrbelic
Lange 2013 (teduer)
Mistral Large 2433 (Har))
Duwee 5-718 inttuct (Sabbald) See
30x
Ouert 5-70: onthact MMddaan
Geniii- w-w.------------ Corbbs NO
Sentis: d. SE ant AND (Paduatt Dee
Servicl 1.55 With 460 (HArE) 1014
boasciing Dop
Stop
Poass ling Soop
Reacziing og Step
Koip
Natso La www.331-77B4eee Symbboll
Netrl 2- E-TERENE (Meckur)) Serv
Hetta 2-1-723330000000 entrrct
Mac Serteeelloodd (yymbal
Hioita Sechhl Macker)
244
Pessor Stop
Rassiring Stea
Kaasziing Step
Deasc Ding Seip
Reaucting
Raszzing
Peassting
Cower 2-784 (Mackur)
Quen' inttrct Stand
therae 2.1 Entrre
Matte Llema LiB.iiitrreett
Figure 14. Accuracy decay with context length for different models
29

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
M. Common Error Analysis
In this section, we collect and analyze representative examples errors made by Llama-3. 1-0e-....-----tsu-ue Instructt and Qwen-2.5-7B: Instruct. We present ten examples, labeled from 10. The following table summarizes common error categories.
Summary of Common Error Types
Error Type
Index of Examples
Confuses two similar but distinctive concepts in the question Hallucinates concept not mentioned in the question Gets distracted by unnecessary variables in the question Misinterprets the relationship mentioned in the question
(I), (8), (10)
(2), (4), (9)
(3)
(5), (6), (7)
Table Common error types observed and their corresponding example indices.
Example 1: 1i-ma--3.-8B-Insttuuce
Error Description: Llama-3.1 1.1-B- --n-stutttt confuses two similar concepts within deduction: "Total number of adult animals in Mayer Aquarium" and "Total number of newborn animal children in Mayer Aquarium", which are very distinct concepts.
QUESTION
Problem:
The average number of newbern children per adult crow in South Zoo equals plus the sum of the total number of newborn animal children in Mayer Aquarium and the nunber of adult eagle in Mayer Aquarium. The number of adult eagle in Mayer Aquarium equals the number of adult owl Mayer Aquarium, The number of adult parrot Zoo equals 4, The average number of newborn children per adult owl in Jefferson Circus equals the sum of the total number of newborn animal children in Mayer Aquarium, the total number of adult animals in Mayer Aquarium, and the average number of newborn childres per adult owl in Mayer Aquarium, The number of adult owl Mayer Aquari equals 3. The number of adult eagle in Jefferson Circus equals 4 times the sum of the number of adult parrot in South Zoo, the number of adult owl in Bundle Ranch, and the average number of newborn children per adult owl in Bundle Ranch. The average number of newborn children per adult eagle in Jefferson Circus equals 4. The nunber of adult crow in South Zoo equals the average number of newborn children per adult cagle in Mayer Aquarium. The average number of newborn chiléren per adult parrot in South 8oo equals 3. The number of adult owl in Bundle Ranch equals 2 plus the number of adult parrot in South Zoo. The average number of newborn children per adult crow in Jefferson Circus equals the average number of newborn children per adult owl in Jefferson Circus. The number of adult owl in Jefferson Circus equals 4. The average number of newborn children per adult owl in Bundle Ranch equals the difference between the number of adult owl in Bundle Ranch and the number of adult parrot in South Zoo. The average number of newborn children per aqult owl in Mayer Aquarium equals 3. The average number of newborn children per adult eagle in Mayer Aquarium equals the total number of adult animals in Mayer Aquarium. The number of adult crow in Jefferson Circus equals the average number of newborn children per adult owl in Bundle Ranch.
South
Question: What is the average number newborn children per adult eagle in Mayer Aquarium?
GROUND TRUTH SOLUTION
Define adult owl in Mayer Aquariun as B; so = 3.
Define adult eagle in Mayer Aquarium y1 so y 3.
Define total number of adult animals in Mayer Aquarium as N; 80 y 6. Define average number of newborn children per adult eagle in Mayer Aquarium as k; so k N = 6.
Answer: 6.
30

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
LLAMA-3.1-8B--NSTRIC OUTPUT
to solve this problem, we need to break down into smaller steps and use the given information to calculate the average number of newborn children per adult eagle in mayer aquarium.
step calculate the total number of newborn animal children in mayer aquarium.
from the problem statement, we know that the average number of newborn children per adult owl in mayer aquarium equals 3. since the number of adult owl in mayer aquarium equals the total number of newborn animal children in mayer aquarium equals 3 9.
step 2: calculate the total number of adult animals in mayer aquarium.
from the problem statement, eagle in mayer aquarium equals the total number of adult animals in mayer aquarium. let's call the total number of adult animals in mayer aquarium as x. then, the average number of newborn children per adult eagle in mayer aquarium equals
know that the average number newborn children per adult
step 3: calculate the average number newborn children per adult eagle in mayer aquarium.
from the problem statement, we know that the average number of newborn children per adult eagle in mayer aquarium equals the total rumber of adult animals in mayer aquarium. therefore, we can write an equation:
=
this equation true for any value of x, however, we need to find value of x. step 4: use the information from the problem statement to find the value of x. from the problem statement, we know that the average number of newborn children per adult eagle in mayer aquarium equals the total rumber of newborn animal children in mayer aquarium. we already calculated the total number of newborn animal children in mayer aquarium as 9. therefore, we can write an equation:
x 9
now we have a value for step 5: calculate the average number of newborn children per adult eagle in mayer aquarium.
since x 9, the average number of newborn children per adult eagle nayer aquarium equals 9.
answer: 9
Example 2: Llama-3.1-B--Instrueet
Error Description: Llama-3. 1---------nnnttee makes up a concept that isn't appeared in the question statement, "Adult Parrot Hamilton Farm" is introduced in the response, which is not mentioned in the question. The newly introduced concept cannot be computed based on the question, since it not mentioned there, and Llama-3.1-8B- InS-trrutttt stuck at computing
QUESTION
Problem: The average number of newbern children per adult blue jay in Bundle Ranch equals 2. The number of adult parrot in Bundle Ranch equals The number of adult blue jay in Bundle Ranch equals times the average number of newborn chiléren per adult blue jay in Bundle Ranch. The number of adult eagle in Bundle Ranch equals 2 times the average number of newborn children per adult blue jay in Bundle Ranch. The number of adult parrot in South Zoo equals 4 times the sum of the average number of newborn children per adult eagle in Hamilton Farm, the number Farm, and the average number of newborn children per adult eagle in Hamilton Farm. The average number of newborn children per adult cagle in Hamilton Farm equals the number of adult eagle in Bundle Ranch. The number of adult eagle in Hami Farm equals The average number of newborn children per adult parrot Bundle Ranch equals the total number of adult animals in Hami lton Farm. The number of adult eagle
adult eagle Hamilton
31

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
in South Zoo equals 1. The average number newborn children per adult parrot in South zoo equals the average number of newborn children per adult parrot in Bundle Ranch. The average number of newborn children per adult eagle in Bundle Ranch equals 3 plus the average number of newborn children per adult parrot in Bundle Ranch. The average number of newborn children per adult eagle in South 200 equals the sum of the nunber of adult blue jay in Bundle Ranch, the average number of newborn children per adult blue jay in Bundle Ranch, the average number of newborn children per adult parrot in Bundle Ranch, and the number of adult parrot in Bundle Banch. Question: What is the average nunber of newborn children per adult eagle in Bundle Ranch?
GROUND TRUTH SOLUTION
Solution: Define adult eagle Hamilton Farm as H; so
Define total number of adult animals in Hamilton Farm as wi so w = H 3. Define average number of newborn children per adult parrot in Bundle Ranch as A; so A = === 3. Define average number of newborn children per adult eagle in Bundle Ranch as a; U ==A= 3; a = + 3 6. Answer: 6.
OUTPUT
to find the average number of newbern children per adult eagle in bundle ranch, we need to follow the given information step by step.
step 1: the number of adult blue jay in bundle ranch equals 2 times the average number of newborn children per aduit blue jay in bundle ranch.
let's denote the average number of newborn children per adult blue jay in bundle ranch
x. then, the number of adult blue jay in bundle ranch 2x.
step 2: the number of adult eagle in bundle ranch equals 2 times the average number of newborn children per adult blue jay in bundle ranch.
since the average number of newborn children per adult blue jay in bundle ranch is x, the nunber of adult eagle in bundle ranch is 2x.
step 3: the average number of newborn children per adult eagle in bundle ranch equals 3 plus the average number of newborn children per adult parrot in bundle ranch.
let's s denote the average number of newborn children per adult parrot in bundle ranch as
y. then, the average nunber of newborn children per adult eagle in bundle ranch is 3 + y.
step 4: the average number of newborn children per adult parrot in bundle ranch equals the total number of adult mnimals in hami Iton farm.
we need to find the total number of adult animals in hami farm. to do this, we need find the number of adult eagle in hami Iton farm and the number of adult parrot in hamilton farm.
step 5: the number of adult eagle in hamilton farm equals 3.
the average number of newborn children per adult eagle in hamilton farm equals the number of adult eagle in bundle ranch.
since the number of adult eagle in bundle ranch 2x, the average nunber of newborn children per adult cagie in hamilton farm is also 2x.
step 6: the average number of newborn children per adult parrot in bundle ranch equals the total number of adult animals in hamilton farm. the total number of adult animals in hamilton farm is the sum of the nunber of adult eagle in hamilton farm and the number of adult parrot in hamilton farm. since the number of adult eagle in hamilton farm is 3, and the average number of newborn children per adult eagle in hamilt
on farm is 2x, the number of adult parrot in hami Iton farm is (2x 3) / y.
step 7: the number of adult parrot in south zoo equals 4 times the sun of the average number of newborn children per adult eagle in hamilton farm, the number of adult eagle in hamilton farm, and the average number of newborn children per adult eagle in hamilton farm.
the sum of the average number of newborn children per adult eagle in hamilton farm, the nunber of adult eagle in hamilton farm, and the average number of newbern children per adult cagle in hamilton farm is 2x 3 2x 4x + 3. therefore, the number of adult parrot in south 200 is (4x + 3).
step 8: the average number of newborn children per adult parrot south zoo equals the average number of newborn children per adult parrot in bundle ranch.
32

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
the average number newborn children per adult parrot bundle ranch y. therefore, the average number of newborn children per adult parrot in south 200 is also y. step 9; the average number of newbern children per adult eagle in south 200 equals the sun of the number of adult blue jay in bundle ranch, the average number of newborn children per adult blue jay in bundle ranch, the average number of newborn children per adult parrot in bundle
ranch, and the number of adult parret in bundle ranch.
the sum of the number of adult blue jay in bundle ranch, the average number of newborn children per adult blue jay in bundle ranch, the average number of newbern children per adult parrot in bundle ranch, and the number of adult parrot in bundle ranch 2x + x 2x 4x x 5x y.
Example 3:
Error Description: Llama-3. 1--B--..------- 1-8B-Instruct already compute the correct answer. However, when it re-evaluating itself, it decides that needs to use unnecessary variables into the calculation by introducing un-mentioned relationship from the question, and Llama-3. 1ama--311-30 1-11-8B-nnsttee arrives at the wrong result.
QUESTION
Problem:
The average number of newborn children per adult blue jay in Oakridge Riverside equals 2. The average number of newborn children per adult parrot in Cedar Valley equals 1 plus the total number adult animals in Oakridge Riversice. The number of adult parrot in Cedar Valley equals the number of adul: blue jay Cedar Valley, The number of adult blue jay in Oakridge Riverside equals the average number newborn chi chind per adult blue jay in Oakridge Riverside. The average number of newborn children per adult blue jay in Cedar Valley equals 4. The number of adult blue jay in Cedar Valley equals times the sum of the average number newborn children per adult blue jay in Oakridge Riverside, the total number of adult animals in Oakridge Riverside, and the number of adult blue jay in Oakridge Riverside. Question: What is the average number of newborn children per adult parrot in Cedar Valley?
GROUND TRUTH SOLUTION
Define average number newborn children per adult blue jay in Oakridge Riverside
so I 2. Define adult blue jay in Oakridge Riverside as Vi so
2. Define
total number of adult animals in Oakridge Riverside as K; so K = 2. Define average number of newborn children per adult parrot in Cedar Vallcy as 0; h = K 2;
0 + 1 2 3. Answer:
LLAMA-3. 1-0--33.--BB-NSSTUCC OUTPUT
to solve this problem, we need to break down into smaller steps and use the given information to calculate the required quantitics.
*+step 1: calculate the number of adult blue jays in oakridge riverside.:
from the problem statement, we know that the number of adult blue jays in oakridge riverside equals the average number of newborn children per adult blue jay in oakridge riverside, since the average number of newborn children per adult blue jay in oakridge riverside equals 2, we can conclude that the number of acult blue jays in oakridge riverside is aLso 2. **step 2: calculate the total number of adult animals in oakridge riverside*s
we already know that the number of adult blue jays in oakridge riverside is 2. sincs there is no other type of animal mentioned for oakridge riverside, we can assume that the total number of adult animals in oakridge riverside is also 2. **step 3: calculate the average number of newborn children valley**
adult parrot in cedar
from the problem statement, we know that the average number newborn children per adult parrot in cedar valley equals 1 plus the total number of adult animals in oakridge riverside. we already calculated that the total number of adult animals in oakridge riverside is 2. therefore, the average number of newborn children per adult parrot in cedar valley equals
3.
**step 4: calculate the number of adult parrots cedar valley**
33

---

GSM-co: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
from the problem statement, we know that the number adult parrots in cedar valley equals the number of aduit blue jays in cedar valley, we don't t know the number of adult blue jays in cedar valley yet, so we' 11 come back to this later.
Asstp 5: calculate the number of adult blue jays in cedar valley**
from the problem statement, we know that the number of adult blue jays in cedar valley equals 3 times the sum of the average number of newborn children per adult blue jay in oakridge riverside, the total number of adult animals in oakricge riverside, and the number of adult blue jays in oakridge riverside. we already calculated that the average number of newborn children per adult blue jay in oakridge riverside is 2, the total number of adult animals in oakridge riverside is 2, and the number of adult blue jays in oakridge rivardide is 2. therefore, the number of adult blue jays in cedar valley equals 3 (2 + 2 2) = 3 3.... 6 18.
**step 6: calculate ate the number of adult parrots in cedar valley.*
from the problem statement, we know that the number of adult parrots in cedar valley equals the number of adult blue jays in cedar valley. we already calculated that the number of adult blue jays in cedar valley is 18. therefore, the number of adult parrots in cedar valley is also 18.
Astepp 7: calculate the average number of newborn children per adult parrot in cedar valley**
from the problem statement, we know that the average number newborn children per adult parrot in cedar valley equals 1 plus the total number of adult animals in oakridge riverside. we already calculated that the total number of adult animals in oakridge riverside is 2. therefore, the average number of newborn children per adult parrot in cedar valley equais
2 =2=3 3.
however, we need to re-evaluate this step. the average number of nexborn children per adult parrot in cedar valley is actually given by the problem em statement as 1 plus the total number of adult animals in oakridge riverside. but we also know that the number of adult parrots in cedar valley equals the number of adult blue jays cedar valley, which is 18. therefore, the average number of newborn children per adult parrot in cedar valley equals 1 + (total number of adult animals in oakridge riverside) + (number of adult blue jays in cedar valley / number of adult parrots in cedar valley) (average number of newborn children per adult blue jay in cedar valley average number of newborn children per adult parrot in cedar valley). we already calculated that the total number of adult animals in oakridge riverside is 2, the number of adult blue jays in cedar valley is 18, and the average number of newborn children per adult blue jay in cedar valley is 4. therefore, the average number of newborn children per adult parrot in cedar valley equals 2 (18/18)
(4 - 3) = 1 ++= 4 answer: 4
Example 4: LJama-3.1-8B-Insttee
Error Description: Llama-3.1-88 1eeeaa3.--BB--nseee --8trunstt make up a concept "adult bear Pine Ridge" that isn't mentioned in the question. The response cannot compute the correct answer from a concept that cannot be computed which introduced earlier in the response.
QUESTION
Problem:
The average number of newbern children per adult bear in Maple Creek equals 4. The number of adult bear in Beverly Forest equals 2. The average number of newborn children per adult bear in Beverly Forest equals 3. The average number of newborn children per adult wolf f in Pine Ridge equals the number of adult bear in Cedar valley. The number of adult wolf in Pine Ridge equals 4. The number of adult racoon in Pine Ridge equals 3 times the number of adult wolf in Pine Ridge. The number of adult wolf in Maple Creek equals 4. The average number of newborn children per adult racoon in Beverly Forest equals the sum of the number of adult racoon in Beverly Forest, the average nunber of newborn children per adult bear in Beverly Forest, and the number of adult wolf in Maple Creek. The average number of newborn children per adult wolf in Maple Creck equals 4 times the sum of the average number of newborn children per adult bear in Beverly Forest and the average number of newborn children per adult bear in Cedar Valley. The number of adult wolf in Beverly Forest equals the sun of the total number of adult animals in Pine Ridge and the number of adult wolf in Pine Ridge. The number of adult racoon in Beverly Forest equals the average number
34

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
newborn children per adult bear Maple Creek. The average number newborn children per adult racoon in Pine Ridge equals the total number of adult animals in Maple Creek. The number of adult bear in Maple Creek equals the number of adult bear in Cedar Valley. The number of adult bear in Cedar Valley equals 2. The average the
nunber of newborn children per adult bear in Cedar Valley equals the average number of newborn children per adult bear in Beverly Forest and the average number of nowborn children per adult bear in Maple Creak. The average number of newborn children per adult wolf in Beverly Forest equals the average number of newborn children per adult bear in Maple Creek. Question: What is the total number of adult animals in Pine Ridge?
GROUND TRUTH SOLUTION
Define adult wolf Pine Ridge as vi so v= 4. Define adult racoon Pine Ridge as R₁
= v = 4; so R = =r= 12. Define total number of adult animals in Pine
Ridge as b; so b ==ve 4 12 = 16. Answer: 16.
OUTPUT
to solve this problem, need to break it down into smaller steps and the given information calculate the total number of adult animals in pine ridge. step 1: calculate the number of adult bear in cedar valley.
the number of adult bear in cedar valley equals 2.
step 2: calculate the average number of newborn children per adult bear in cedar valley. the average number of newborn children per adult bear in cedar valley equals the sum of the average number newborn children per adult bear in beverly forest and the average number of newborn children per adult bear in maple creek.
let's denote the average number of newborn children per adult bear in beverly forest as x and the average number of newborn children. per adult bear in maple creek 4,
the average number of newborn children per adult bear in cedar valley step 3: calculate the average number newborn children per adult wolf pine ridge. the average number of newborn children per adult wolf in pine ridge equals the number of adult bear in cedar valley.
+
the average number of newborn children per adult wolf in pine ridge step 4: calculate the number adult wolf in pine ridge. the number of adult wolf in pine ridge equals 4. step 5: calculate the number of adult racoon in pine ridge.
the number of adult racoon in pine ridge equals 3 times the number of adult wolf in pine ridge.
the number of adult racoon in pine ridge = step 6: calculate the total number of adult animals in pine ridge. the total number of adult animals in pine ridge equals the sum of the number of adult wolf, the number of adult zacoon, and the number of adult bear in pine ridge. however, the number of adult bear pine ridge is not given, so we need to calculate it. stop 7: calculate the average number of newborn childrea per adult racoon in pine ridgo. the average number of newborn children per adult racoon in pine ridge equals the total number of adult animals in maple creek.
however, the tctal number adult animals in maple creek is not given, calculate it.
need
step 8: calculate the total number of adult animals in maple creek. the total number of adult animals in naple creek equals the sum of the number of adult bear, the number of adult wolf, and the number of adult racoon in maple creek.
however, the number of adult raccon in maple creek is not given, so we need to calculate it.
step 9: calculate the number of adult racoon in maple creek.
the number of adult racoon in maple creek equals the average number of newborn children per adult bear in maple creek. however, the average number of newbern children per adult bear in maplc creck equals 4, stop 10: calculate the number of adult racoon in maple crock. the number of adult racoon in maple creek 4
step 11: calculate the total number of adult animals in maple creek. the total number of adult animals raple creek equals the of the number of adult bear, the number of adult wolf, and the number of adult racoon in maple creek.
35

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
the number of adult bear in maple creek equals the number of adult bear in cedar valley,
which equals 2.
the number of adult wolf in maple creek equals 4.
the total number of adult animals in maple creek
4 + = 10
step 12: calculate the average number of newborn children per adult racoon in pine ridge. the average number of newborn children per adult racoon in pine ridge equals the total number of adult animals in maple creek.
the average number of newborn children per adult racoon in pine ridge = 10
step 13: calculate the number of adult racoon in pine ridge.
the number of adult racoon in pine ridge equals 3 times the number of adult wolf in pine ridge.
however, we already calculated the number of adult racoon in pine ridge in step 5, which equals 12.
step 14: calculate the total number of adult animals in pine ridge. the total number of adult animals in pine ridge equals the sum of the number of adult
wolf, the number of adult racoon, and the number of adult bear in pine ridge.
the number of adult wolf in pine ridge equals 4.
the number of adult racoon in pine ridge equals 12.
however, the number of adult bear pine ridge is not given, so we need to calculate it. step 15: calculate the average number of newborn children per adult wolf in beverly forest.
the average number of newborn children per adult wolf in beverly forest equals the average number of newborn children per adult bear in mapl
creek.
the average number of newborn children per adult wolf in beverly forest 4 step 16: calculate the number of adult wolf in beverly forest.
the number of adult wolf in beverly forest equals the sum of the total number of adult animals in pine ridge and the number of adult wolf pine ridge.
however, we already calculated the total number of adult animals in pine ridge in step 14, which equals 20.
the number of adult wolf in beverly forest = 20 + 4 24
step 17: calculate the number of adult bear in beverly forest.
the average number of newborn children per adult bear in beverly forest equals 3. the total number of newborn children per adult bear in beverly forest equals the average number of newborn children per adult bear in beverly forest times the number of adult bear in beverly forest.
however, the number of adult bear in beverly forest not given, so we need to calculate it.
step 18: calculate the number of adult bear in beverly forest. the number of adult bear in beverly forest equals 2.
step 19: calculate the total number of newborn children per adult bear in beverly forest. the total number of newborn children per adult bear in beverly forest equals the average nunber of newborn children per adult bear in beverly forest times the number of adult bear in beverly forest.
the total number of newborn children per adult bear in beverly forest step 20: calculate the average number of newborn children per adult bear in cedar valley. the average number of newborn children per adult bear in cedar valley equals the sum of the average number of newborn children per adult bear in beverly forest and the average number of newborn chiedren per adult bear in maple creek.
the average number of newborn children per adult bear in cedar valley + 7 step 21: calculata the number of adult bear in maple creek.
the number of adult bear in maple creek equals the number of adult bear in cedar valley, whien equals 2.
step 22: calculate the total number of newborn children per adult bear in maple creek. the total number of newborn children per sdult bear in maple creek equals the average nunber of newborn children per adult bear in maple creek times the number of adult bear in maple creek.
the total number of newborn children per adult bear in maple creek = = 4
step 23: calculate the number of adult racoon in beverly forest.
the number of adult racoon in beverly forest equals the average number of newborn children per adult bear in maple creek.
the number of adult racoon in beverly forest =
step 24: calculate the average number of newborn children per adult racoon in beverly forest.
36

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
the average number of newborn children per adult racoon beverly forest equals the sum of the number of adult racoon in beverly forest, the average number of newborn
children per adult bear in beverly forest, and the number of adult wolf in maple creek.
the average number of newborn children per adult racoon in beverly forest = + + 11
step 25: calculate the total number of newborn children per adult racoon in beverly forest.
the total number of newborn children per adult racoon in beverly forest equals the average number of newborn children per adult racoon in beverly forest times the nunber of adult racoon in beverly forest.
the total number of newborn children per adult racoon in beverly forest 11 * 44 step 26: calculate the total number of newborn children per adult racoon in pine ridge. the total number of newborn children per adult racoon in pine ridge equals the average nunber of newborn children per adult racoon in pine ridge times the number of adult racoon in pine ridge.
however, we already calculated the average number of newborn children pez adult racoon in pine ridge in step 12, which equals 10.
the total number of newborn children per adult racoon in pine ridge 10 12 120 step 27: calculate the total number of newborn children per adult wolf in pine ridge. the total number of newborn children per adult wolf in pine ridge equals the average number of newborn children per adult wolf in pine ridge
times the number of adult wolf in pine ridge.
the total number of newborn children per adult wolf in pine ridge = =
step 28: calculate the total number of newborn children per adult bear in pine ridge. the total number of newborn children per adult bear in pine ridge equals the average number of newborn childres per adult bear in cedar valley times the number of adult bear in cedar valley.
the total number of newborn children per adult bear pine ridge step 29: calculate the total number of newborn children in pine ridge. the total number of newborn children in pine ridge equals the sum of the total number of newborn children per adult wolf, the total number of newborn children per adult racoon, and the total number of newborn children per adult bear in pine ridge. the total number of newborn children stop 30: calculate the total number of adult animals in pine ridge. the total number of adult animals in pine ridge equals the sum of the number of adult
14
pine ridge
+ 120 14 142
wolf, the number of adult racoon, and the number of adult bear in pine ridge.
however, the number of adult bear in pine ridge is not given, so we need to calculate it. step 31: calculate the number of adult bear in pine ridge.
the number of adult bear in pine ridge equals the total number of newborn children per
adult bear in pine ridge divided by the average number of newborn children per adult bear in cedar valley.
the number of adult bear in pine ridge step 32: calculate the total number of adult animals in pine ridge. the total number of adult animals in pine ridge equals the sum of the number of adult wolf, the number of adult racoon, and the number of adult bear in pine ridge. the total number of adult animals in pine ridge answer: 18
7
12 2 18
Example 5: Llama-3.1-8B-Instruut
1. Instrrut misinterpret the condition given in the question. The question states that "average number of newborn children per adult eagle in Maple Creek" equals + "average number of newborn children per adult eagle in Cedar Valley". However, during the responding, -lama-3.1-8B-Insttttte newborn children per adult eagle in Maple Creek" as the sum of "average number of newborn children per adult eagle in Cedar Valley" and "average number of newborn children per adult owl in Cedar Valley" which is different from what the question mentions.
Error Description:
compute "average number of
QUESTION
Problem: The average number of newbern children per adult owl in Cedar Valley equals 4. The average number of newborn children per adult eagle in Maple Creek equals 2 plus the average number of newborn children per adult eagle in Cedar Valley. The average nunber of newborn children per adult eagle Cedar Valley equals 2. The number of
37

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
adult eagle Maple Creek equals plus the sum of the average number of newborn children per adult owl in Cedar valley and the number of adult owl in Cedar Valley. The number of adult owl in Oakridge Riverside equals 2 plus the difference between the average number of newborn children per adult eagle in Cedar Valley and the average number of newborn children per adult owl in Oakridge Riverside, The average nunber of newborn children per adult owl in Oakridge Riverside equals the sum of the average number of newborn children per adult eagle in Maple Creek and the averago number of newborn children per adult eagle in Cedar Valley, The number of adult eagle in Cedar Valley equals 3 times the difference between the average number of newborn children per adult eagle in Cedar Valley and the number of adult owl in Cedar Valley, The number of adult ow in Cedar Valley equals the average number of newborn chi per adult owl in Cedar Valley. Question: What is the average number of newborn children per adult owl in Oakridge Riverside?
GROUND TRUTH SOLUTION
Define average number newborn children per adult eagle in Cedar Valley ui 80 2. Define average number of newborn children per adult eagle Maple Creek Vi M u = 2; so v = 2 + M = - = 4. Define average number of nowborn children per adult owl in Dakridge Riverside as qi so q = = 6. Answer: 6.
LLAMA-3.1-88B I--SSRUUCT'SSS OUTPUT
to solve this problem, we need to carefully follow the given information and use it to find the average number of newborn children per dult owl in cakridge riverside.
here are the steps:
1. let's dencte the average number of newborn children per adult owl in cedar vallsy as a, the average number of newborn childrer. per adult eagle cedar valley as b, the average number of newborn children per adult eagle in maple creek as c, the number of adult eagle in maple creek d, the number of adult owl in cedar valley nunber of adult owl in oakridge riverside f, and the average number of newborn children per adult owl in oakridge riverside as g.
the
2. from the problem statement, we know that a 4.
3. also know that
4. furthermore, we are given that = b + a = 2 + 4 6.
5. the problem also states that d 2 (a + e).
6. we are also given that c a 4.
7. substituting the value of into the equation for d, get d = (4 + 4) 10.
8. the problem also states that f 2 (b g). =
9. we are also given that g +
10. suostituting the value of into the equation for f, we get f = - 12 - 8) -4.
however, since the number of adult animals cannot be negative, we need to re-eval uate the equation for f.
11. let's re-evaluate the equation for g: g -
2
12. now, let's s re-evaluate the equation for f: f 2 (b - g) 2 + (2 8) 2
-4. however, this is still not correct.
13. let's re-evaluste the equation for f again: f = 2 + (b - g) 2 (2 9) =
-4. however, this is still not correct.
14. let's re-evaluate the equatien for f once more:
(b g) 2 (2 8)
= -4. however, this is still not correct.
15. ist's re-evaluate the equation for f again: f = 2 + (b - g) 2 (2 8) = 2 - 6
-4. however, this is still not correct.
16. let's re-evaluate the equation for f once more: f (b g) 2 (2 8)
however, this is still not correct.
38

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
17. let's re-evaluate the equation for again: f (b g) 2 12
-4. however, this is still not correct.
18. let's re-evaluate the equation for f once more: f - 2 + (b. 9) 2 12 8) 2
= -4. however, this is still not correct.
19. l re-evaluate the equation for again: f
(b g) 2
(2
81
2
-4. however, this is still not correct.
20. l re-evaluate the equation for f once more: f 2 + (b - g) 2
(2
8) = 2
= -4. however, this is still not correct.
21. let's re-evaluate the equation for again: f (b g) 2 (2
81
-4. nowever, this is still not correct.
22. s re-evaluate the equation for once more:
(b
9)
2
(2 8)
- -4. however, this is still not correct.
23. let's re-evaluate the equation for again: f 2 (b g)
(2 8)
-4. nowever, this is still not correct.
24. let's re-evaluate the equation for f once more:
(b g)
2 (2 8)
= -4. however, this is still not correct.
25. let's re-evaluate the equation for f again: f = 2 + (b - gg) 2 (2 8) = 2
-4. however, this is still not correct.
26. let's re-cvaluate the equation for once more:
g)
2
(2 8) 2
-4. however, this is still not correct.
27. let's re-evaluate the equation for f again: f = 2 + (b - g) 2 (2 8) = 2
-4. however, this is still not correct.
28. let's re-evaluate the equation for f once
(b
g)
2 (2 8)
-4. nowever, this still not correct.
29. s re-evaluate the equation for f again: f 2 + (b - g)
2 (2
8) = 2 -
-4. however, this is still not correct.
30. let's re-evaluate the equation for f once more: f 2 (b.
g)
2
(2 8)
-4. however, this still not correct.
31. let's re-evaluste the equation for again: f
(b g)
2 (2
8)
-4. however, this is still not correct.
32. l re-evaluste the equation for f once more: f 2 (b g)
2 (2 8) = 2
-4. however, this is still not correct.
33. let's re-evaluate the equatien for again:
(b g) 2 (2
81
-4. however, this is still not correct.
34. let's re-evaluate the equation for f once more: f 2 + (b - g) 2
(2 8) = 2 -
= -4. however, this is still not correct.
35. let's re-evaluate the equation for again: f (b g) 2 (2
-4. nowever, this is still not correct.
36. let's re-evaluate the equation for f once more: f 2 + (b - g) 2 (2 8) =
= -4. nowever, this is still not correct.
37. let's re-evaluate the equation for again: f
(b g) 2 (2
8)
-4. however, this is still not correct.
38. let's re-evaluate the equation for f once more: f + (b - g) 2
(2 8) = 2
= -4. however, this is still not correct.
39. let's re-evaluate the equation for f again: f 2 + (b - g) 2 (2
8) =
-4. however, this is still not correct.
40. let's re-cvaluate the equation for f once more:
(b g)
2 (2 8)
= -4. however, this is still not correct.
41. let's re-evaluate the equation for f again: f = 2 + (b - g) 2 (2 8) =
-4. however, this is still not correct.
42. let's re-evaluate the equation for once more:
g)
(2 8)
-4. however, this is still not correct.
43. let's re-evaluate the equation for f again: f - 2 + (b - g) 2. (2 8) - 2
-4. however, this is still not correct.
44. let's re-evaluate the equation for once
g)
2 (2 8)
-4. however, this still not correct.
45. let's re-evaluate the equation for f again: f + (b - g) 2 (2 8) =
-4. however, this is still not correct.
46. let's re-evaluate the equation for f once more: f (b g) 2 (2 8)
-4. however, this is still not correct.
47. let's re-evaluate the equation for again: f
(b g) 2 (2
8)
4. however, this is still not correct.
48. let's re-evaluate the equation for f once more: f = 2 (b - 2 (2 8) = 2
-4. however, this is still not correct.
49. let's re-evaluate the equation for f again:
(b g) 2 (2 8)
39

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
however, this is still not corrsct.
50. let's re-evaluate the equation for f once more: f + (b - 9) 2 12 8)
- -4. however, this is still not correct.
51. s re-evaluate the equation for f again: f 2 + (b - g)
12
8) =
-4. however, this is still not correct.
52. l re-evaluate the equation for É once more: f
(b
g)
2
(2 8)
= -4, however, this is still not correct.
53. l re-evaluate the equation for f again: f = 2 (b - g) 2 (2
8) = 2
-4. nowever, this is still not correct.
54. let's re-evaluate the equation for f once more:
(b
g)
2 (2 8)
-4. however, this is still not correct.
55. l re-evaluate the equation for f again: f 2 + (b - g) 2 (2
8) - 2
-4. however, this is still not correct.
56. s re-evaluate the equation for once
g)
2 (2 8)
-4. however, this still not correct.
57. l re-evaluate the equation for f again: f = 2 + (b - g) 2 (2
8) =
-4. however, this is still not correct.
58. let's re-evaluate the equation for f once
f
(b 9)
(2 8)
-4, however, this still not correct.
59. let's re-evaluate the equation for again: f + (b g) 2 (2
8) 6
-4. however, this is still not correct.
60. let's re-evaluate the equation for f once more: f 2 (b g) 2 (2 8)
-4. however, this is still not correct.
61. s re-evaluate the equation for f again:
(b g) 2 (2
81
-4. nowever, this is still not correct.
62. let's re-evaluate the equation for f once more: f 2 (b 9) 2 (2 8)
-4 however, this still not correct.
63. let's re-evaluate the equation for f again: f = 2 + (b - g) 2 (2 8) = 2
-4. however, this is still not correct.
64. let's re-evaluate the equaticn for once more:
9) 2
12 8)
= -4. however, this is still not correct.
65. let's re-evaluate the equation for f again: f 2 (b g) 2 12 8)
4. however, this is still correct.
66. let's re-evaluate the equation for f once more: f 2 + (b 9) 2
(2 8) = 2
= -4. nowever, this is still not correct.
67. let's re-evaluate the equation for again:
(b g) 2 12
-4. however, this is still not correct.
68. let's re-evaluate the equation for f once more: f 2 (b g) 2
(2 8) = 2
-4. however, this is still not correct.
69. iet's re-evaluate the equation for f again: f + (b - g) 2 (2 8) 6
4. however, this is still not correct.
70. let's re-evaluate the equation for once more: f
g)
8)
-4. however, this is still not correct.
71. s re-evaluste the equation for & again: f = 2 + (b - g)
12
9) = 2
-4. however, this is still not correct.
72. let's re-evaluate the equation for f once more: f + (b - 9) 2 (2 8) = 2 6
= -4. however, this is still not correct.
73. let's re-evaluate the equation for f again: f + (b g)
(2
8)
-4. however, this is still not correct.
74. let's re-evaluate the equation for f once more:
+ (b g)
2 (2 8)
= -4. however, this is still not correct.
75. let's re-evaluate the equation for f again: f = 2 + (b - g) 2 (2 8) = 6
-4. however, this is still not correct.
76. let's re-evaluate the equation for É once more:
(b g) 2 (2 8)
40

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
-4, however, this still not correct.
77. let's s re-evaluate the equation for f again: f + (b - 9) 2 12 8) = -
-4. however, this is still not correct.
78. let's re-evaluate the equation for f once more: f 2 + (b g)
2
(2
8) = 2
-4. however, this is still not correct.
79. s re-evaluate the equation for É again: f
(b g)
2
(2
8)
-4. however, this is still not correct.
80. s s re-evaluate the equation for f once morc: f = 2 (b - g) 2
(2
8) = 2
-4. nowever, this is still not correct.
81. l re-evaluate the equation for f again:
(b g) 2 (2
8)
-4. however, this is still not correct.
82. s re-evaluate the equation for f once more: f 2 + (b - g) 2
(2
8) - 2
-4. however, this is still not correct.
83. let's re-evaluate the equation for again: f (b g) 2 12
81
-4. however, this is still not correct.
84. l re-evaluate the equation for f once more: f
Example 6: Qwen-2.5-7B-Instruue
Error Description: Owen-2.5-77-Innstru uses number of "adult eagle in Bundle Ranch" and "adult parrot in Bundle Ranch" to compute the "total number of adult animals in Hamilton Farm", which shows that Qwen-2.5-7B-Insse misinterpret the relationships in the question.
QUESTION
Problem: The average number of newbern children per adult blue jay in Bundle Ranch equals 2. The number of adult parrot in Bundle Ranch equals 2. The number of adult blue jay in Bundle Ranch equals times the average number newborn children per adult blue jay in Bundle Ranch. The number of adult eagle Bundle Ranch equals times the average number of newborn children per adult blue jay in Bundle Ranch. The number of adult parrot in South Zoo equals 4 times the sum of the average number of newborn children per adult cagle in Hamilton Farm, the number of adult cagle in Hamilton Farm, and the average number of newborn children per adult cagle in Hamilton Farm. The average number of newborn children per adult eagle in Hamilton Farm equals the number of adult eagle in Bundle Ranch. The number of adult eagle in Hamilton Farm equals 3. The average nunber of newborn children per adult parrot in Bundle Ranch equals the total number of adul: animals in Hamilton Farm. The number of adult eagle in South Zoo equals 1. The average number newborn children per acult parrot in South Zoo equals the average number of newborn children per adult parrot in Bundle Ranch. The average number of newborn children per adult eagle in Bundle Ranch equals 3 plus the average number of newborn children per adult parrot in Bundle Ranch. The average number of newborn children per adult eagle South 8oo equals the sum of the number of adult blue jay in Bundle Ranch, the average number of newborn children per adult blue jay in Bundle Ranch, the average number of newborn children per adult parrot in Bundle Ranch, and the number of adult parrot in Bundle Ranch. Question: What is the average nunber of newborn children per adult eagle in Bundle Ranch?
GROUND TRUTH SOLUTION
Define adult eagle in Hamilton Farm as H; so 3. Define total number of adul: animals in Hami lton Farm as w; so w = н 3. Defire average number of newborn children per adult parrot in Bundle Ranch as A; so A = w = 3. Define avorage number of newborn children per adult eagle in Bundle Ranch as a; PAC 3; =
6.
Answer:
11PNN-2.5-7B-INSTTIICE OUTPUT
let's s break down the problem step by step to find the average number of newborn children. per adult eagle in bundlc ranch.
1. *averag number of newborn children per adult blue jay in bundle ranch:**
given: 2
2. **number of adult blue jay bundle ranch:*

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
given: times average number newborn children per adult blue jay bundle ranch
calculation: (2 Atimes - 41)
3. Anmmber of adult eagle in bundle ranch:*4
given: 2 times the average number of newborn children per adult blue jay in bundle ranch
calculation: (2 \times
41
4. **average number of newborn children
adult eagle in hamilton farm:**
given: the number of adult eagle in bundle ranch calculation:
numbber of adult eagle in hami ttn farm:**
given: 3
6. *ummbe of adult parrot in south 200:**
given:
the average number of newborn children per adult eagle in hamilton farm, the namber of adult eagle in hamilton farm, and the average number of newborn children per adult eagle in hamilton farm
4 times the
calculation: (4 \times (4 +
4)
\times 11 = 44))
7. +vverage number of newborn children per adult parrot in bundle ranch:** given: the total number
adult animals in hamilton farm
calculation: total number of adult animals hamilton farm number adult blue jay + number of adult eagle + number of adult parrot in bundle ranch 8))
calculation: (4 4
*vverage number of newborn children per adult eagle in bundle ranch:**
given: 3 plus the average number of newborn children per adult parrot in bundle ranch calculation: (33 + 8 11))
9. *average number of newborn children per adult parrot in south 200:**
given: the average number of newborn children per adult parrot in bundle ranch calculation:
10. *vveraee number of newborn children per adult eagle south zoo:..
given: the sum of the number of adult blue jay in bundle ranch, the average number of newborn children
adult blue jay in bundle ranch, the average number of newborn children per adult parrot in bundle ranch, and the number of adult parrot in bundle ranch calculation: 144 + 141)
Example Qwen-225-77-Innstrucl
Error Description: (wwnn2255-7B-Insteee cannot understand the essential relationships introduced the question. directly guess a number which isn't n't correct.
QUESTION
Problem:
The number of adult wolf Pine Ridge equals 4 plus the average number of newborn children per adult wolf in Pine Ridge. The number of adult aeer in Pine Ridge equals 3. The number of adult wolf in Oakricge Riverside equals the sum of the the average
average aumber of newborn caildren per adult deer in Cedar Valley nunber of newborn children per adult deer Cedar Valley equals 4. The average number of newborn children per adult deer Pine Ridge equals 4 plus the average number of newborn children per adult wolf in Cedar valley. The average number of newborn children per adult wolf in Oakridge Riverside equals The average number of newborn children per adult wolf in Cedar Valley equals 2. The average number of newborn children per adult deer in Oakridge Riverside equals the difference between the average number of newborn children per adult deer in Pine Ridge and the average number of newborn children per adult wolf in Codar valley. The average nunber of newborn children per adult deer in Cedar Valley equals the number of adult deer in Cedar Valley. The average number newborn children per adult wolf in Pine Ridge equals times the average number of newborn children per adult deer in Pine Ridge. The number of adult wolf in Cedar Valley equals the sum of the namber of adult deer in Cedar Valley and the average number of newbern children per adult deer in Cedar Valley. The number of adult deer in Oakriage Riverside equals the average number newborn children per adult wols Oakridge Riverside, Question: What is the average number of rexborn children per adult wolf in Pine Ridge?
Pine Ridge. The number of adult deer in
42

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
GROUND TRUTH SOLUTION
Define average number of newborn children per adult wolf in Cedar Valley as Tj s0 I 2. Define average number of newborn children. per adult deer in Pine Riage as a; L T 2; so = 4 = 4 2 6. Define average number of newborn children per adult wolf in Pine Ridge as S;
6; so S = 2 = 12. Answer: 12.
OWEN-2.5-7B-INSTRUCE OUTPUT
let's s break down the problem step by step to find the average number of newborn childrer. per adult wolf in pine zidge.
1. +*number of adult wolves in pine ridge:**
let V w_lltext (pine ridge)1 V) be the number adult wolves in pine ridge.
according to the problem, ( (piteet ridge)]
\text(average number of
newborn children per adult wolf in pine ridge) V.
let \< (_txe (wolf, pine ridge)) \) be the average number of newborn children per adult wolf in pine ridge.
therefore, \ x_(Atex ridge))
e_(ltext (wolf, pine ridge)} M..
2. **number of adult deer pine ridge:**
the problem states that the number of adult deer in pine ridgo is 3.
therefere, ' d_ (pine ridge)) = V).
3. numbbrr of adult wolves in oakridge riverside:**
let ( w_(teet (oakridge riversida) \) be the number of adult wolves in oakridge riverside.
according to the problem, (( w_ (wt.eettttakkiiiaa
riverside) \text (average number of newborn children per adult deer in cedar valley! + \text (average number of newborn children per aduit deer in pine ridge) V).
let \ e ltett (deer, cedar valley)) V) be the average number of newborn children per adult deer in cedaz valley.
let \ c_(Xtext (deer, pine ridge)) \) be the average number of newborn children per adult deer in pine zidge.
therefore, \ (c(ttexttcakrrdd riverside] c_(Ntext (deer, cedar valley)) c_(\text (deer, pine ridge]] V).
4. **number of adult deer in cedar valley:**
the problem states that the number of adult deer in cedar valley is 4. therefere, \ d_(teett (cedar valley))
\).
5. aveeraee number of newoorn children per adult deer in pine ridge:** according to the problem, A (Llltext/dees pine ridge)) = 4 e_eetext (wolf, cedar valley)) \)
1et \ (_(ttex(wwolf adult wolf in cedar valley.
cedar valley)) \) be the average number of newborn children per
e /teetee//woo 6. avveraee number of newborn children per adult wolf in cedar valley:** the problem states that \i c_l\text (wolf, cedar valley]) = A). therefere, \ e_(\text (welf, cedar valley! = V).
therefore, \ e_/ltext
pine ridge)
cedar valley]] V).
7. aveerae number of newborn children per adult deer in cedar valley:..
the problem states that the average number of newborn children per adult deor in cedar valley equals the number of adult deer in cedar valley.
therefore, \ e_(\text (deer, cedar valley) 4 \).
8. *average number of newborn children per adult deer in oakridge riverside:**
according to the problem, \ c_ltext (deer, oakridge riverside; e_ /deer, pine ridge]] - c_ (ttext cedar valley! V).
therefore, \ e_i\text (deer, oakridge riversidell (4 + c_iltext twolf, cedar valley))) [_iNtex(woo1f cedar valley!] - V.. number of adult wolves in cedar valley:** according to the problem, \ (Xtext valley)) (Llttex(cceda valley)) + /deer, cedar valleyl] V).
therefore, \( w_/\text (cedar valley)) = + 4 \).
10. **average number of newborn children per adult wolf in oakridge riverside:s
the problem states that the number of adult deer in oakridge riverside equals the average number of newborn children per adult wolf in oakridge riverside. therefore, \( d_fttett (oakridge riverside)] c_(ooxt therefore, \( (wo1,, cakridge riverside]; = \).
oakridge riverside)) V).
**average number of newborn children per adult wolf in pine ridge: we know \ v_(Atext (pine ridge)] 4 itett (wolf, pine ridge] A).
43

---

GSM-co: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
we need to find ( (wel, pine ridge) V).
from the equation ( (_(tte ridge) - + (_(Atext pine zidge)) M), we can substitute ( [_:\text ridge]] V) with the value we need to find. pine ridge:; V..
we know A "Littextppine ridge]] = 4 e_iltext we need to find \ (_lltex pine ridge)) V).
given the equation A w_/ltext (pine ridge)) = 4 + e_(teext (wolf, pine ridge)] M, and knowing \ w_(Ntoxt zidge)} V must be a positive integer, we can solve for \ c_ittext (wolf, pine ridget: V.
since (( (_(Ntex (pine ridge)) \) must be positive integer, and we have no other constraints, we can assume ( e_tetett/ (xl,,, pine ridgel] 2 A).
therefore, the average number of newborn children per adult wolf in pine ridge is X 2 V). answer: 2
Example 8: Qwen-225-7B-Insttuuet
Error Description: Owen-2.5-7BB-Isttt confuses "adult crow in Oakridge Riverside" with "average number of newborn children per adult crow in Oakridge Riverside", leading to incorrect result.
QUESTION
Problem:
The average number of newbern children per adult crow in Pine Ridge equals 4. The average number of newborn children per adult owl in Cedar Valley equals the total number of adult animals in Oakridge Riverside. The average number of newborn children per adult blue jay in Pine Ridge equals the sum of the number of adult owl in Pine Ridge, the average number of newborn children per adult eagle in Beverly Forest, the number of adult blue jay in Cedar Valley, and the number of adult owl in Cedar
Valley. The average nunber of newborn children per adul: eagle in Oakridge Riverside equals 4 plus the average number of newborn children per adult blue jay in Oakridge Riverside. The average number of newborn children per adult crow in Beverly Forest equals the number of adult eagle in Oakridge Riverside. The number of adult blue jay in Pine Ridge equals 2. The number of adult blue jay in Beverly Forest equals plus the average number of newborn children per adult blue jay in Beverly Forest. The average number of newborn children per adult crow in Cedar Valley equals 4 times the sun of the number of adult blue jay in Beverly Forest and the number of adult eagle in Oakridge Riverside. The number of adult owl in Beverly Forest equals 2 times the sun of the average number of newborn children per adul: owl in Oakrioge Riverside and the average number of newborn children per adult eagle in Cedar Valley. The number of adult owl in Cedar Valley equals 3 plus the di fference between the average number of newborn children per adu't blue jay in Beverly Forest and the number of adult eagle in Oakridge Riverside, The average number of newborn children per adult blue jay in Beverly Forest equals 2. The number of adult blue jay in Cedar Valley equals 4 plus the difference between the number of adult blue jay in Beverly Forest and the number of adult owl in Cedar Valley. The average number of newborn children per adult owl in Oakridge Riverside equals 4. The number of adult owl in Pine Ridge equals 2, The average number of newborn children per adult blue jay in Cedar valley equals 2. The average number of newborn children per adult crow in Oakridge Riverside equals the sun of the average nunber of newborn children per adult blue jay in Beverly Forest and the total number of adult animals in Oakridge Riverside. The number of adult crow in Cedar Valley equals the total number of adult animals in Pine Ridge. The average nunber of newborn children per adult owl in Pine Ridge equals 4. The average number of newborn children per adult owl in Beverly Forest equals 3 plus the number of adult owl in Cedar Valley. The number of adult eagle in Oakridge Riversice equals the average number of newborn children per adult blue jay in Beverly Forest. The number of adult crow in Oakridge Riverside equals 3. The average number of newborn children per adult eagle in Cedar Valley equals the sum of the total number of adult animals in Oakridge Riverside, the average number of newborn children per adult blue jay in Beverly Forest, and the average number of newborn children per adult blue jay in Beverly Forest. The nunber of adult crow in Pine Ridge equals 2. The number of adult owl in Oakridge Riverside equals 3. The number of adult eagle 2 plus the average number of newborn children per adult blue jay in Beverly Forest. The number of adult cagie in Beverly Forest equals the number of adult owl in Pinc Ridge. The average number of newborn children per adult eagle in Beverly Forest equals 3 times the nunber of adult sagle in Cedar Valley. The number of adult blue jay in Oakridge Riverside de equals 1. The average number of newborn children per adult
Cedar Valley equals
44

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
blue jay Oakridge Riverside equals plus the average number of newborn children per adult crow in Pine Ridge. The number of adult crow in Beverly Forest equals 2. Question: What is the average number of newborn children per adult eagle in Oakridge Riverside?
GROUND TRUTH SOLUTION
Define average number of newborn children per adult in Pine Ridge so 4.
Define average number of newborn children per adult blue jay Oakridge Riverside as W; U a 4; so П = 4 == 4 8. Define average number of newborn children per adult eagle in Cakridge Riversi de as N; v w 8, so N + 12. Answer: 12.
OWWN-2.5-7B-INSTRUCCE OUTPUT
let's break down the problem step by step find the average number of newborn childrer. per adult eagle oakridge riverside.
1. **numbor of adult blue jay in pine ridge:**
M
Atext (number of adult blue jay in pine ridge)
A]
2.
*average number of newborn children per adult crow in pine ridge:
\[
\text laverage number of newborn children per adult crow in pine ridgel
\
*average number of newporn children per adult blue jay in beverly forest:**
\I
\text (average number of newbern children per adult blue jay in beverly forest)
\]
nnmbeer of adult blue jay beverly forest:..
M[
ttex (number of adult blue jay in beverly forest] = 2 + =
\]
5. **number of adult blue jay in cedar valley:**
M
ttet (number of adult blue jay in cedar valley) = 4 (4 (3 (2 -41) = +
A]
6.
*xnumber of adult owl in cedar valley:*x
M
\text (number of adult owl cedar valley) + (2 4)
\I
7. **vverage number of newborn children adult owl in cedar valley:**
M
\text (average number newbern children per adult owl cedar valley)
\]
8.
number of adult owl in beverly forest:**
M
teex (number of adult owl in beverly forest: Atimes 3) Atimes V
9.
*vveraee number of newborn children per adult oxl in oakridge riverside:**
MI
Neex (average number of newbern childres per adult owl in oakridge riverside)
\]
10.
*ummee of adult owl in pine ridge:**
\I
\text (number of adult owl in pine ridge) =
VJ
11.
*enumber of adult eagle in cedar valloy:**
M[
\text (number of adult eagle in cedar valley) = +
VJ
12.
**veragee number of newborn childres per adult eagle cedar valley:**
45

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
Atext number of newborn children per adult eagle cedar valley)
\]
13.
nummber of adult eagle in oakridge riverside:..
Atext (number of adult eagle in cakridge riverside)
V)
14.
**average number of newborn children per adult blue jay in oakridge riverside:**
\I
\text (average number of newborn children per adult blue jay in oakridge ziverside)
15.
**veragee number of newborn chien per adult crow in oakridge riverside:**
\l
(text (average number of newborn children per adult czow in oakridge ziverside] = 11
\
16.
+number of adult crow in cedar valley:*4
\text (number of adult crow cedar valley)
\J
17.
**number of adult crow in oakridge riverside:
MI
Atext (number of adult erow oakridge riverside)
V
18.
**average number of newborn chi Idren per adult crow in pine ridge:*
MI
Ntext (average number of newborn children per adult crow in pine ridge)
\
19.
*avvragee number of newborn children per adult blue jay in beverly forest:**
Aext (average number of newborn children per adult blue jay in beverly forest)
20.
*verage number of newborn children per adult cagle in cedar valley:**
Atext (average number of newborn children per adult eagle in cedar valley]
\]
21. **veragee number newoorn children per aduit eagle in beverly forest:**
\I
\text laverage number of newbern children adult eagle in beverly forest] 3 Atimes
4 12
22. average number of newborn children per adult owl in beverly forest:**
\I
\text (average number of newborn children adult owl in beverly forest) 3 + =
\
23. **average number newborn children per adult blue jay oakridge riverside:**
\[
\text (average number of newbern children adult blue jay in oakridge riverside) =
+ -
\]
24. **veragee number of newborn chi Idren per adult eagle oakridge riverside:**
\[
\text (average number of newbern children adult eagle in oakridge riverside)
3
\
answer: 11
Example 9: Qwen-2.5-7B-IIntrrus
Qwen-2.5-7B. ---Itruttt hallucinates "adult wolf in Maple Creek" that isn't mentioned in the question, which leads incorrect result.
Error Description:
46

---

GSM-co: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
QUESTION
Problem:
The number of adult fox in Reverly Forest equals the sum of the average number of newborn chi Idren per adult wolf in Beverly Forest, the average number of newborn children per adult wolf in Cedar Valley, and the average number of newborn children per adult fox in Beverly Forest. The average number of newborn chilaren per adult fox in Pine Ridge equals 3. The average number of newborn children per acult deer in Oakridge Riverside equals 4. The average number of newborn children per adult fox in Beverly Forest equals the sum of the average number of newborn children per adult wolf in Beverly Forest, the average number of newborn children per acult deer in Pine Ridge, and the average number of nowborn children per adult bear in Cedar Valley, The number of adult deer in Cedar Valley equals 1 plus the sum of the average number of newborn children per adult deer in Pine Ridge, the average number of newborn children per adult wolf in Beverly Forest, and the average number of newborn children per adult wolf in Cedar Valley. The average number of newborn children per adult bear in Cedar Valley equals 3 times the average number of newborn children per adult wolf in Beverly Forest. The average number of newborn children per adult bear in Beverly Forest equals the sum of the average number of newborn children per adult wolf in Cedar valley, the total number of adult animals in Cedar valley, the average number of newborn children per adult fox in Beverly Forest, and the total number of newborn animal children in Pine Ridge. The number of adult deer in Oakridge Riverside equals 3. The number of adult bear in Cedar Valley equals the total number of adult animals in Cedar Valley. The nunber of adult deer in Maple Creek equals 3. The number of adult bear in Beverly Forest equals the sum of the average number of newborn children per adult bear in Pine Ridge, the avorage number of newborn children per adult fox in Beverly Forest, the average number of nowborn children per adult wolf in Beverly Forest, and the number of adult deer in Cedar Valley. The average number of newborn children per adult deer in Cedar Valley equals the sum of the total number of newborn animal cdren in Mapie Creek, the total number of adult animals in Maple Creek, and the number of adult deer in Maple Creek. The average number of newborn children per adult wolf in Beverly Forest equals 1. The average number of newborn children per adult wolf in Cakridge Riverside equals the average number of newborn children per adult deer in Pine Ridge. The number of adult bear Maple Creek equals times the number of adult deer in Maple Creek. The number of adult wolf in Cedar Valley equals 4 times the average number of newborn children per adult deer in Pine Ridge, The number of adult wolf in Beverly Forest equals the sun of the average number of newborn children per adult bear in Maple Creek, the average number of newborn children per adult wolf in Cedar Valley, and the average number of newborn children per adult deer in Pine Ridge. The number of adult bear in Pine Ridge equals 4, The number of adult deer in Pine Ridge equals 1. The average number of newbern children per adult deer in Pine Ridge equals the difference between the average number of newborn children per adult wolf in Reverly Forest and the average number of newborn childre per adult bear in Cedar Valley. The number of adult wolf in Oakridge Riverside de equals 3 times the sum of the number of adult bear in Maple Creek, the total number of adult animals in Maple Creek, and the number of acult deer in Maple Creek. The average number of newborn children per adul= bear in Pine Ridge equals the nunber of adult wolf in Cedar Valley. The number of adult fox in Pine Ridge equals 4. The average number of newborn children per adult deer in Maple Creek equals 4, The average number of newborn children per adult bear in Maple Creek equals 3. The average number of newborn children per adult wolf in Cedar Valley equals 1, Question: What is the total number of adult animals in Maple Creek?
GROUND TRUTH SOLUTION
solution: Define adult deer in Maple Creck as M; so M- 3. Define adult bear in Maple Creek as h; U = M = 3; so h = 3 *
=3 3 3 3. Define total number of adult animals
in Maple Creek as E; so H h 3 + 9 = 12. Answer: 12.
OWEN-2.5-7B-INSTRUCE OUTPUT
let's break down the problem step by step to find the total number of adult animals in maple creek.
1. **aeerage number of newborn children per adult deer in pine ridge:ve
[[
47

---

\text (average number of newborn children per adult deer in cedar valleyl = 3 \times \text number of newborn children per adult wolf in beverly forest) - GSM-co: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
text (average number newborn children per adult deer pine ridge)
\]
2. * *average number of newborn children per adult wolf in beverly forest:**
[[
Aeex (average number newborn children per adult wolf in beverly forest)
3. **average number of newborn children per adult bear in cedar valley:**
M
\text (average number of newbern children per adult bear in cedar valley) Atimes
stextlavered number of newborn children per adult wolf in beverly forest)
Atimes
\]
4. **average number of newborn children per adult fox in beverly forest:**
\I
ttet laverrae number newborn children per adult fox in beverly forest
Itext (average number of newborn children per adult wolf in beverly forest) +
Itext (average number of newborn children per adult deer in pine ridge) + (averaee number of newborn children per adult bear in cedar valley)
M]
M
+ 1+1++==
M]
5. *number of adult deer in cedar valley:**
VI
\text (number of adult deer in cedar valley) = -1 \text laverage number of newborn children per adult deer in pine ridge) \text (average number of newborn children per adult wolf in beverly forest) + \text (average number of newborn children per adult wolf in cedar valley)
A]
M
++++1+1=
A]
6. numbbr of adult bear cedar valley:**
M[
\tex (number of adult bear in cedar valley) = \text number of adult animals in cedar valley)
\]
7.
**average number of newporn children per adult deer in cedar valley:.x
\I
\times
\]
8. wnmmber of adult deer in oakridge riverside:**
M
text (number of adult deer in oakridge riverside)
V]
9.
knumber of adult bear in beverly forest:**
M
\text (number of adult bear in beverly forest Atext number of newborn chi Idren per adult bear in pine ridge) Atext (average number newborn children per adult fox in beverly forest) + \text (average number of newborn children per adult wolf in beverly forest) + \text (number of adult deer in cedar valley)
M
M
+ 5
1]
10.
nnmber of adult wolf in cedar valley:**
Ntext (number of adult wolf in cedar valley) \times (text (average number newborn. children per adult deer in pine ridge) \times
\)
11. *enumber of adult wolf in beverly forest:*>
\I
Atext (number of adult wolf beverly forest) Atext (average nunber of newborn
48

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
children per adult bear maple creek) Atext (average number newborn children per adult wolf in cedar valley) Ntext number of newborn children per adult deer in pine ridge)
A
=++++==5
V
12.
**average number of newborn children per adult deer in maple crecki**
AI
Atext (average number of newborn children per adult deer in maple ereek) \text (total number of newborn animal children in maple creek) + itext[tota number of adult animals in maple creek) + (texben of adult deer in maple creek)
A]
AI
+ 10
V
13.
**average number of newborn children per adult bear in pine ridge:**
MI
Atext (average number of newborn children per adult bear in pine ridge! Atext (number of adult wolf in cedar valley)
\)
14.
**number of adult bear in pine ridge:**
MI
Atext (number of adult bear pine ridge)
V
15.
*enumber of adult deer in pine ridge:**
M
Atext (number of adult deer pine ridge)
\
16.
*raverage number of newborn children per adult deer in pine ridge:**
\I
Atext (average number of newborn children per adult deer in pine ridge) = \text (average number of newborn children per adult wolf in beverly forest) (average number of newborn children per adult bear in cedar valley)
A]
A
V
17.
numbeer of adult wolf in oakridge riverside:**
\I
Atext (number of adult wolf in cakridge riverside; Atimes (Ntext/number of adult bear maple creek; Atext (total number of adult animals maple creek) Ntext (number of adult deer in maple creek))
AJ
A[
\times (3 3) \times 27
18.
numbbr of adult fox in pine ridge:**
\I
Ntext (number of adult fox in pine ridge) -
19.
**average number of newborn chi 1dren per adult bear maple creek:
\[
\text (average number of newbern children adult bear in maple creek) =
\
20. **verag number newborn children per adult wolf cedar valley:**
M[
Atext (average number of newborn children per adult wolf in cedar valley) 1
21.
**mmber of adult wolf in beverly forest:..
\[
\text (number of adult wolf in beverly forest] (average number of newborn children per adult bear in maple creek) \text (average number of newborn children per adult wolf in cedar valley] \text (average nunber newborn children per adult deer in pine ridge!
49

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
3
-+1+1-5
22.
*tttal number of adult animals in maple creek:*.
AI
Atext (total number of adult animals in maple creek) = \text (number of adult bear in maple creek) + Atext inmbber of adult decr in maple creck) + \text (number of adult wolf in maple creek)
A
A
+ 3
A]
Example 10: Qwen-225-7B-Instruuc
Error Description: Confuses "adult owl in Pine Ridge" with "average number of newborn children per adult owl in Pine Ridge", which leads to incorrect result.
QUESTION
Problem:
The number of adult cagle in Cakridge Riverside equals the sum of the number of adult owl in Cakridge Riverside and the average number of newborn children per adult eagle in Oakridge Riverside, The average number of newborn children per adult ow1 in 0akri dge Riverside equals plus the difference between the average number of newborn children per adult ow! Pine Ridge and the number of adult ow1 Oakridge Riverside. The number of adult owl in Oakridge Riverside equals 2. The average number of newborn children per adult eagle in Pine Ridge equals 3 times the sun of the average number of newborn children per adult eagle in Oakridge Riverside, the average nunber of newborn children per adult owl in Fine Ridge, and the number of adult eagle in Oakridge Riverside. The average number of newborn children per adult owl in Pine Ridge equals 2. The nunber of adult owl in Fine Ridge equals the average number of newborn children per adult cagle in Oakridge Riverside, The average number of newborn children per adult eagle in Oakridge Riverside equals The number of adult eagle in Pine Ridge equals the average number of newborn children per adult eagle Oakridge Riverside. Question: What is the total number of adult animals in Pine Ridge?
GROUND TRUTH SOLUTION
Define average number newborn children per adult eagle in Cakridge Riverside g7 g Define adult owl in Pine Ridge as ji so 3, Define adult eagle Pine
Ridge as r; so r g = 3. Define total number of adult animals in Fine Ridge as t; 80
t - + 3 6. Answer: 6.
QWEN-2.5-7B INSTRUCT'S OUTPUT
let's break down the problem step by step.
1. **number of adult
oakridge riverside:w*
given: (22)
2. **average number of newporn children adult ewl in cakridge riverside:** 1et (xt) be the average number of newborn children per adult t owl in oakridge riverside.
given: \(x = + ('text (average number of newborn children per adult owl in pine ridgel - 2)
let \yy) be the average number of newborn children per adult owl in pine ridge. (y 2)
sc, Mx
y\)
3. **average number of newborn children per adult eagle oakridge riverside:** given: \(3))
4. *number of adult eagle in oakridge riverside:*.
let (z)) be the number of adult eagle in oakridge riverside.
given: \zz
5))
5. **average number of newborn children per adult eagle pine ridge:wv
50

---

GSM-oo: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?
let Ma\) be the average number of newborn children per adult eagle in pine ridge.
given:
(a 3 Atimes (3 + + 5) 3 Atimes 10 301)
6. number of adult owl in pine ridge:A*
given: (21)
7. avverage number of newoorn children per adult owl in pine ridge:..
given: (21)
8. **number of adult eagle in pine ridge:**
given: (33)
now, we need to find the total number of adult animals in pine ridge.
number adult eagle in pine ridge: (3V
number of adult owl pine ridge: (22)
total number of adult animals in pine ridge:
AI + - \] 5
51


Reviews:
[{"review_id": "CMsvC1ijWU", "reviewer": "Reviewer_6KRZ", "summary": "This paper points out issues in existing long-context reasoning benchmarks and addresses the issues by designing new benchmark called GSM-Infinite. \nThe main method they use is to construct computation graphs for problems in GSM8K and generate question-answer pairs with user-definable question difficulty measured by reasoning steps. \nThe main findings is that LLMs' performance decays with reasoning complexity, addition of noise. They struggle more on backward thinking tasks and benefit from more inference steps. \nFor model performance, they find that reasoning models do better than non-reasoning LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Three missing citations**\n\nGreatly appreciate the comments. We acknowledge that more discussion of previous multi-hop commonsense reasoning benchmarks **will be added in the paper later version**. But these three papers address different problems from ours. \n\nFirstly, HotpotQA [2] and MusiQue [3] are on short-context 2 to 4-hop commonsense reasoning. Also, huge human labor is required to collect, limiting scaling up on quantity or the reasoning complexity. Also, these two are converted to long-context problems incorporated in RULER ([2]) and LOFT ([2] and [3]), which are cited and heavily studied (Section 2, page 3). The added noise in two can be effectively filtered by RAG, inapt for LC-LLM evaluation. \n\nSecondly, CLUTRR [1], though aired before LLMs, similarly generates problems of family relationship deduction from graphs but with 3 big distinctions. (1) Despite similarity in methods, CLUTRR is heavily tailored to family relation deduction, especially specialized to kinship graphs, severely limited by scope. \n(2) Human annotation is essential to making CLUTRR, while GSM-Infinite relies completely on software, no LLM nor humans. No guarantee of the absolute correctness of the labels provided by the generator is a crucial drawback, as humans make mistakes. Specifically, we found that a minor part of the dataset is mislabeled. One example below. \n\n*Label: sister\nQuestion: Given the story, please provide the relationship between two selected people. Story: [James] bought a new dress for his daughter [Lynn]. [Theodore] and his son [Wesley] went to look at cars. [Theodore] ended up buying the Mustang. [James] took his grandson [Theodore] to the baseball game. Question: What is the relationship between 'Wesley' and 'Lynn'?*\n\nThe corrected label is Wesley is the grand nephew of Lynn.     \n\n(3) Having either LLM or Human in the loop also leads to huge difficulty in scaling up the reasoning complexity. Below, we evaluate the 2to10 test set given by CLUTRR with various LLMs. \n| Models   | Accuracy |\n|---|---|\n| Llama 3.1 8B Instruct | 0.64     |\n| GPT-4o  | 0.7      | \n\nLack of scaling prevents the evaluation of the full spectrum of behaviors. GSM-Infinite generates op=200 problems (and more if needed) to capture the full spectrum of SOTA models' reasoning performance. \n\n**RAG clarification** \n\nWe follow the convention of context-level RAG method [1]. We solely use the input context as the data store (no external source). We segment the input text into chunks (roughly a sentence) using NLTK package. The retriever first rates each chunk based on its relevance to the query, and then the top-k scoring chunks are selected as the input to the decoder model. k is determined by the 2k budget from our experiments. \n\n[1] Lewis, P. Retrieval-augmented generation for knowledge-intensive nlp tasks. (2020) \n\n**Dataset generation with Graph Clarification** \n\nDuring generation, no human/LLM is in the loop. “One-shot” generation is irrelevant to our settings. Each problem, regardless of context length, is generated based on one computational graph (not multiple graphs). Natural language problems directly mapped from the graph using our developed generator software. For a detailed explanation and reference, please look at (Figure 6, page 6) and (Section 3.1, 3.2, page 3-5). \n\n**Forward Reverse Observation** \n\nThe difference between forward and reverse problems isn’t the arithmetic operations. Referring to (Section 4.3, page 6), the explicit plus, minus, multiply, and divide operations appear in both forward and reverse problems. **The difference is the implicit operation/relationships contained.** The implicit addition and multiplication are displaying class-instance and hierarchical instance dependency, respectively. Problems in forward logic only contain these two implicit operations - you can notice that in the computation graph (Figure 6, page 6), the more concrete and detailed instances are first computed, and then, using them collectively, more general and summarizing variables are computed. However, reverse problems contain the implicit minus and division operations. Given the value for the abstract variables, the LLM needs to compute the unknown instance variables. Here is a simplified example. \nForward Problem - #pigs in Zoo is 2. #sheeps in Zoo is 4. Assume the animals not mentioned are of 0 quantity. What is the #total animals in Zoo? \nSolution: We go from #pigs in Zoo (instance) -> #sheeps in Zoo (instance) -> total animals (abstract). \nReverse Problem - #pigs in Zoo is 2. #Sheep in Zoo is non-zero. #total animals in Zoo is 6. #sheep in Zoo? \nSolution: We go from total animals (abstract) -> #pigs in Zoo (instance) -> #sheeps in Zoo (instance). \nWe show a detailed study in (Appendix E, page 18). We hypothesize that models' usual strength in forward stems from that humans naturally write in the constructive logic ordering, or forward, more, leading to naturally less training data for reverse logics.", "claims_and_evidence": "Claims they made: \n1) LLM Performance Degradation Can be Modeled Using Sigmoid Function\n2) Reverse Problems are Harder to Solve for LLMs\n3) LLMs' performance degrade as the context grows long and as noise is added. \nAll of them are supported by experimental results.", "methods_and_evaluation_criteria": "Their evaluation method for different LLMs with different context windows make sense.", "theoretical_claims": "There is no theoretical claims in this paper.", "experimental_designs_or_analyses": "See Questions.", "relation_to_broader_scientific_literature": "I think this paper is related to the long-context LLM reading comprehension literature and also multi-hop reasoning benchmarks.", "essential_references_not_discussed": "This paper does not provide discussion on multi-step reasoning literature. \nThere are several classical benchmarks which should be included in the paper: \n\n[1] Sinha, Koustuv, et al. \"CLUTRR: A diagnostic benchmark for inductive reasoning from text.\" arXiv preprint arXiv:1908.06177 (2019).\n[2] Yang, Zhilin, et al. \"HotpotQA: A dataset for diverse, explainable multi-hop question answering.\" arXiv preprint arXiv:1809.09600 (2018).\n[3] Trivedi, Harsh, et al. \"♫ MuSiQue: Multihop Questions via Single-hop Question Composition.\" Transactions of the Association for Computational Linguistics 10 (2022): 539-554.", "other_strengths_and_weaknesses": "The authors did a good job explaining details e.g. how to build computation graph for reasoning problems including how to handle different operations and how to add noise. \nHowever, I feel that it lacks a good explanation on how the whole pipeline works. See Questions.", "other_comments_or_suggestions": "Some of the references are repeated or having inconsistent formatting. \ne.g. \n\nKamradt, G. Needle in a haystack - pressure testing llms, 2023. URL https://github.com/gkamradt/LLMTestNeedleInAHaystack/tree/main.\nis repeated. \n\nChieh. et al. 2024a and 2024b are the same.", "questions_for_authors": "It is possible that I missed the relevant part, but I am confused about the RAG setting: what is treated as a single \"article\"? Is it a single graph? \nWhy is it needed that that LLMs have 8k context minimum? Are you putting all the available graphs and generate questions at one shot? \nWhat is the intuition behind \"reverse problems are harder to solve\"? After the problems are generated properly using the computation graph, shouldn't it be the same for the models to solve dividing and minus math problems as solving the addition and multiply types?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8zSZpll3Sb", "reviewer": "Reviewer_fqRq", "summary": "The paper introduces a new benchmark designed to evaluate the reasoning capabilities of Large Language Models (LLMs) in long and complex contexts.  The benchmark is inspired by the abstraction of GSM-8K problems as computational graphs and introduces a methodology to generate grade-school math problems with infinitely scalable difficulty and context length.  The authors evaluated existing LLMs, revealing a consistent sigmoid decline in reasoning performance as complexity increases, and an inefficiency in performance gains relative to inference computation scaling.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Concern GSM-Infinite not practical** \n\nWe appreciate the reviewer’s concern about GSM-Infinite’s practicality. While GSM-8K has been a standard for evaluating LLM math reasoning, its difficulty has become saturated (Figure 5(a), Page 4), which is why newer models have moved on. GSM-Infinite builds on GSM-8K by abstracting its core operations into graph representations and using graph perturbation to scale both reasoning complexity and context length. Our pipeline is entirely LLM- and human-free, generated independently of GSM-8K (Section 3, Pages 3–5). By scaling up the op=200 and above, we saturate SOTA LLMs and capture their full spectrum of reasoning ability for study. \n\nOn the other hand, recent LLMs often report metrics from MATH or AIME datasets. Although these datasets are of strong reasoning complexity, solving test examples convincingly requires a lot of math-related theorem knowledge in LLMs’ memory. GSM-Infinite offers an alternative reasoning benchmark that is drastically less reliant on LLMs’ memory, while evaluating all LLMs based solely on their reasoning ability of traversing a complex computational graph problem with fine-grained difficulty control for the study. \n\nBecause of space limitations, we hold five examples for op=5 for both Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct in the following link: https://docs.google.com/document/d/1WP3ygB67yNUS-iliSYYjFRY4Ls81yIVcRVxpbTVTZ0o/edit?usp=sharing. \n\nWe select a smaller operation setting with a smaller model to keep the complexity of the problem lower for the practicality of human inspection. Below is a summary of error causes and the corresponding index of the example. \n| Error Type                                                             | Index of the examples     |\n|------------------------------------------------------------------------|---------------------------|\n| Confuses two similar but distinctive concepts in the question          | (1), (8), (10)            |\n| Hallucinate a concept in the question, which the question does not mention | (2), (4), (9)             |\n| Get distracted by unnecessary variables in the question                | (3)                       |\n| Misinterpret the relationship mentioned in the question                | (5), (6), (7)             | \n\n**Generation Throughput** \n\nDuring generation, we strictly constrain the number range in the solution path and disallow negative or floating-point numbers in arithmetic operations. These constraints make generating the Hard subset of GSM-Infinite significantly slower than Symbolic and Medium, as Hard requires heavy use of aggregation, implicit multiplication and addition when op is large—which quickly inflate numbers and reduce flexibility compared to Symbolic, where operations can be mixed more freely. Despite this, our system achieves far superior throughput on the Hard subset compared to any LLM- or human-in-the-loop approach. Using 16 threads on an M3 Pro, we reach 1.1M tokens/s for examples averaging 1k tokens (op=50), a speed that's hard to match with LLM-based generation.\n\n| op  | Throughput (Valid Examples/s) |\n|-----|-------------------------------|\n| 10  | 5510.88                       |\n| 20  | 6747.20                       |\n| 30  | 1919.68                       |\n| 40  | 2228.00                          |\n| 50  | 1149.92                       |\n\nWhen going up the op number, the throughput will go down because the chance that the operation will avoid the strict rejection sampling criteria is decreasing, causing some wasted computation. Just for reference, the much more flexible Symbolic subset can generate problems with 30k examples/s with 16 threads on CPU, since we can have more precise control of every operation in the Symbolic benchmark. We think the need for strong rejection sampling criteria is an implementation challenge, and more careful and more efficient graph control implementation is greatly possible and likely to lead to further improvement in throughput. \n\n|Context Length|Throughput (Valid Example/s)|\n|---|---|\n|Zero noise|6747.20|\n|8K|3896.48|\n|16K|3932.96|\n|32K|3898.08|\n|64K|3917.60|\n|128K|3954.18|\n\nThe above shows throughput for op=20 for increasing context length. The sudden drop in throughput from zero noise to 8k is because when injecting noise to a specific context length requirement, we have to first perform a tokenization step of the zero-noise example and calculate how much noise statement we need. The one extra tokenization causes a slowdown in generation throughput. On the other hand, since we impose far less strict checking on the noise statements, generating noise is highly efficient and parallelizable across multiple threads. We show our generator’s strong context length flexibility.", "claims_and_evidence": "Mostly yes. \n\nThe first remaining concern is that the current dataset fit a rather simple pattern. It is unclear how models' performance would change if optimizing prompt or enabling certain tool use like code execution would change the performance and corresponding conclusion. Some failure mode analysis of a few LLMs is helpful to understand the possible flaws of the dataset.\n\nThe second concern is that it is unclear how the proposed dataset reflects real use case of reasoning models.", "methods_and_evaluation_criteria": "No concerns.", "theoretical_claims": "The authors claimed infite length and mentioned reduced probability of multiple solutions but some formal analysis on the cost of generating a valid question w.r.t. length and the actual probability of such cases should be provided.", "experimental_designs_or_analyses": "Please check above for details.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please check above for details.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QGEg5oA8Bp", "reviewer": "Reviewer_djUT", "summary": "This paper introduces GSM-∞, a synthetic long-context reasoning benchmark generated entirely by an automatic system with fine-grained control over complexity and information density. Specifically, it generates the benchmark by modifying operations like \"+ -\" in the computational graphs of existing benchmarks, which makes the generation method easily scalable. Extensive experiments on this benchmark further leads to several useful findings that are useful for further research in this direction.\n\n## update after rebuttal\nIn the rebuttal, the authors answered my questions and provided an analysis that addresses my previous concerns. Therefore, i increased my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to other reasoning relationship question** \n\nWe want to express our great appreciation to the reviewer for your positive feedback on GSM-Infinite. The question raised is also highly insightful and greatly appreciated. We want to clarify that the goal of GSM-Infinite is to model all the relationships that appear in GSM-8K with graph representation, an abstract way that is easier for manipulation and perturbation for scaling up in reasoning complexity and context length. GSM-Infinite model all explicit operations from plus, minus, multiply, and divide, and the implicit operations with hierarchical instance dependency. For example, “pigs” and “cats” are specific instances under the class “animal”, which GSM-Infinite can model well. (Is that the entailment relationship you are talking about?) \n\nOn the other hand, GSM-Infinite by no means captures all the relationships possible through graph representations. We believe exploring these alternative complex relationships through a graph is also highly interesting and insightful. We will leave these explorations for further complex graph to natural language mappings to future works.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes, I check the correctness of such proofs and claims.", "experimental_designs_or_analyses": "Yes, I check the experiments in the corresponding sections.", "relation_to_broader_scientific_literature": "The key contribution of the paper is related to a certain range of literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The proposed benchmark generation method is directly applied on computational graphs of existing benchmarks, which makes it easily scalable in an automatic way.\n2. The paper is written in a good flow, and experimental results have demonstrated the effectiveness of the proposed benchmark.\n3. Several interesting findings such as the discussion on repeated sampling are presented.\n\nWeaknesses:\n1. There are no obvious weaknesses in this paper, except few typos that are listed below.", "other_comments_or_suggestions": "1. Line 209, ” ->“\n2. Line 431, the reference is not working", "questions_for_authors": "Have the authors tried other reasoning relationships that can be obtained through the graph, like entailment?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wce2UcaOl1", "reviewer": "Reviewer_6Z5g", "summary": "The paper introduces ​GSM-∞, a synthetic benchmark for evaluating long-context LLMs on grade-school math problems. It generates problems with ​controllable complexity (via computation graphs) and ​arbitrary context length (via \"spider topology\" noise). The paper finds that:\n1. ​Sigmoid performance decay as reasoning complexity increases.\n2. Reverse problems (backward reasoning) are harder than forward ones.\n3. ​Exponential inference cost yields only linear performance gains.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Additional figures and tables in the link (anonymous): \nhttps://docs.google.com/document/d/1WP3ygB67yNUS-iliSYYjFRY4Ls81yIVcRVxpbTVTZ0o/edit?usp=sharing (referred to as **Sup**) \n\n**1. No ablation on different graph structures.**\n\nThank you. We agree studying graph structure impact is valuable. We ran Qwen 2.5 7B IT on 40k Symbolic (op=9) to analyze the relationship of graph depth and accuracy. Depth is the max node distance from the root; mean depth is the average across nodes. We categorized samples by max depth (Depth=2 to 6). \n\nResults confirm deeper graphs are harder. Accuracy decreases as depth increases:\n|Depth|2|3|4|5|6|\n|---|---|---|---|---|---|\n|Accuracy|.438|.386|.347|.317|.257|\n\nFurther analysis shows accuracy decreases as *mean depth* increases. When mean depth is similar, max depth has little impact, suggesting overall complexity (mean depth) is the primary factor. A plot confirms this trend across depth settings: **Sup** Part 1. \n\n**2. Limited Analysis between models.**\nBelow data from Hard subset. \nAcknowledged. Based on Table 1 (p7):\n*   **Reasoning Models:** Show significant gains over non-reasoning counterparts with identical parameters/architecture. \n|op|DeepSeek R1|DeepSeek V3|Qwen QWQ 32B Preview|Qwen 2.5 32B Instruct (non-reasoning)| \n|---|---|---|---|---| \n|10|.95|.79|.64|.61| \n|20|.94|.60|.60|.36| \n|30|.95|.36|.26|.12| \n|40|.92|.16|.13|.04| \n|AUC|8573.8|2407.9|2846.2|1399.8|\n*   **Model Size:** Within families, larger models perform better. \n|op|Qwen2.5 72B Instruct|Qwen2.5 7B Instruct|Llama 3.1 70B Instruct|Llama 3.1 8B Instruct|\n|---|---|---|---|---|\n|5|.95|.59|.73|.52|\n|10|.82|.29|.64|.40|\n|15|.76|.18|.49|.19|\n|20|.64|.05|.33|.07|\n|25|.49|.03|.26|.06|\n|30|.37|.00|.04|.02|\n|AUC|2016.375|618.500|1205.250|606.500|\n*   **Architecture:** Current hybrid models (Linear Attention/SSM) lag behind comparable Transformer models. \n|Models|Architecture|ParameterSize|AUC|\n|---|---|---|---|\n|Qwen-2.5-32B-Instruct|Transformer|32B|1405.055|\n|MiniMax-Text-01|Hybrid (Linear Attention)|456BMoE(45.9B/token)|1178.510|\n|Jamba-1.5-Large|Hybrid (SSM)|398BMoE(98B/token)|466.400|\n\n**3. No cost discussions.**\n\nWe compute the cost for Llama-3.1-70B below as a reference. We evaluated 100+ examples per op across 3 subsets (use 100). We evaluated up to 40-55 ops per subset (sampling frequency decreases after op 30). Total zero-noise evaluation (~9300 examples) requires ~9.8M input / ~39.2M output tokens. Using competitive (DeepInfra) API pricing, costs are:\n|Context|Cost|\n|---|---|\n|Zero|$12.94|\n|8K|$5.83|\n|16K|$7.49|\n|32K|$12.60|\n|**Total**|**$30.43**|\n\nLarger/reasoning models cost more. With additional tables and elaboration in **Sup** Part 2., found using strides (evaluating every N ops) up to 4 has minimal impact (<3% AUC change) on results. Reducing samples per op (e.g., 100 or 50) also yields similar AUC, allowing further cost reduction.\n\n**4. Need more detailed graph generation process.**\n\nAcknowledged. We will make sure to add greater and finer step-by-step details about the generation process in the revised version of the paper. \n\n**5. No Model Error Discussion**\n\nWe analyzed errors for Llama-3.1-8B and Qwen-2.5-7B on op=5 problems. Examples available in **Sup** Part 4. \nCommon error types:\n|ErrorType|ExampleIndices(Internal)|\n|---|---|\n|Confuse similar concepts|(1),(8),(10)|\n|Hallucinate concepts not in question|(2),(4),(9)|\n|Distracted by irrelevant variables|(3)|\n|Misinterpret relationships|(5),(6),(7)|\n\n**6. Noise distribution impact & RAG.**\n\nOur ablation (Sec 5.3, Fig 7c/d) shows model performance is robust to noise types, but RAG struggles significantly with our \"Spider Topology\" noise. Our RAG baseline is strong (all-mpnet-base-v2 retriever, Llama-3.1-70B decoder, 2k token budget, strong benchmark performance). RAG fails because it uses contextual (semantic similarity) search. Spider Topology adds irrelevant nodes semantically close to and connected to core graph nodes. RAG's fixed budget means noise can displace relevant chunks. RAG retrievers evaluate chunk relevance contextually, lacking the deep reasoning needed to identify truly necessary information, which attention layers possess. Contextual info alone is insufficient.\n\n**7. Forward vs. Reverse Problems.**\n\nThe difference isn't arithmetic ops, but implicit relationships (Sec 4.3, Fig 6). Forward problems use class-instance/hierarchical dependencies (concrete -> abstract). Reverse problems provide abstract values to find unknown instance values (abstract -> concrete). Most LLMs perform better on forward problems (App E), hypothesized due to training data bias towards constructive logic.\n\nWe fine-tuned Qwen-2.5-500M with 1.3M generated problems (equal forward/reverse). The base model showed forward > reverse initially. Fine-tuning significantly boosted overall performance, with reverse slightly outperforming forward, confirming training data can address this. Detailed data in tables before and after fine-tuning is in **Sup** Part 3.", "claims_and_evidence": "Mostly yes.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes.\nThe paper tests 18 models on zero-noise tasks and 10 on long-context.", "relation_to_broader_scientific_literature": "This paper is related to GSM8K, RULER, and LongBench.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The paper isolates reasoning from retrieval.\n2. The ablation analysis is comprehensive.\n\nWeaknesses:\n1. Lack of ablation study on different graph structures.\n2. Limited Analysis when comparing models.\n3. Insufficient discussion about compute costs.", "other_comments_or_suggestions": "1. The explanation of the computation graph approach can be improved by introducing more detailed breakdowns.\n2. Add more qualitative analysis of model errors.", "questions_for_authors": "1. In section 3.2, how different distributions of noise impact performance? Can a carefully designed RAG system skip it?\n2. How different types of graph structures impact model performance?\n3. The paper finds that LLMs perform worse on reverse problems than forward ones. Whether it is due to lack of training or data in the training set. Can it be effectively improved by providing more similar data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GGNxIXvoK1", "reviewer": "Reviewer_GB4k", "summary": "This paper introduces a new benchmark for testing the performance of LLMs on long-context reasoning.In particular, the authors start from the abstract computational graphs of the GSK-8K problems and develop the GSM-Infinite benchmark.With the constructed benchmark, the authors observe a consistent sigmoidal decline in reasoning performance as the complexity increases.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Concern about Practicality of GSM-Infinite** \n\nWe appreciate the reviewer’s concern about the practicality of GSM-Infinite, as the problem is crucial for understanding why we believe GSM-Infinite is helpful for researchers to use. We want to kindly rebut that first, different from “many synthetic methods for expanding seed questions into long-context reasoning tasks with controllable quality,” GSM-Infinite introduces two important techniques to offer important additional edges over past benchmarks. \n\nFirst, unlike prior work that expands GSM-8K seeds, our test examples are generated entirely from scratch using graph-based generation—no GSM-8K questions or data are used (Section 3, Pages 3–5). GSM-8K is saturated in both quantity and difficulty (Figure 5(a), Page 4), with only ~10 examples per op>8. In contrast, GSM-Infinite offers >500 test cases per op, scaling up to op=200+, saturating SOTA LLMs. \n\nSecond, prior long-context reasoning benchmarks from \"expanding seed\" can often have noise filtered out by retrieval (RAG), making them poor tests of long-context LLMs (Figure 4(a), Page 4). Our graph-based expansion produces problems where noise is indistinguishable from essential info (Figure 4(b), Page 4), rendering RAG ineffective. GSM-Infinite scales both reasoning and context length while being fully LLM/human-free. \n\nMoreover, we believe that GSM-Infinite offers the reasoning community a new tool that evaluates LLMs solely on reasoning with complex graph traversal, without testing them with memorization of subject-specific knowledge. Recent LLMs often report metrics from MATH or AIME datasets, since prior datasets such as GSM-8K have been saturated. Although these datasets are of strong reasoning complexity, solving test examples convincingly requires a lot of math-related theorem knowledge in LLMs’ memory. We show in (Table 1, page 7) that the powerful reasoning models also see their performance decaying to zero as we gradually increase the ops. GSM-Infinite offers an alternative reasoning benchmark that is drastically less reliant on LLMs’ memory, while evaluating all LLMs based solely on their reasoning ability of traversing a complex computational graph problem with fine-grained difficulty control for the study.\n\n**Lacks to infinite dataset** \n\nThank you for this insightful suggestion. Our current implementation, GSM-∞, focuses specifically on modeling the explicit arithmetic operations (+,-,×,÷) and implicit hierarchical relationships found in GSM-8K using computational graphs. The primary goal was to create a scalable benchmark for grade-school math reasoning with controllable complexity and context length.\n\nWhile the graph representation is potentially applicable to a broader range of problems, our current work does not aim to build a universal \"infinite\" dataset covering all reasoning types. Extending this methodology to model other complex structures, like code syntax trees or spatial relationships, is a promising direction for future research.\n\n**Dataset Quality Inspection and Overview** \n\nQuestions were constantly evaluated during development via manual inspection and by feeding them to SOTA LLMs to ensure they were natural, solvable, and LLM-understandable. SOTA models demonstrate high accuracy (>88% for o3-mini on Hard op 30) on zero-noise, lower-op problems, indicating fundamental understandability. \n|op|Forward|Reverse|Score|\n|---|---|---|---|\n|10|1.00|0.96|0.98|\n|20|0.98|0.90|0.95|\n|30|0.94|0.84|0.89|\n\nErrors observed stemmed from model reasoning limitations (e.g., confusing concepts, hallucination, misinterpreting relationships), not ill-formed prompts. We collect examples in https://docs.google.com/document/d/1WP3ygB67yNUS-iliSYYjFRY4Ls81yIVcRVxpbTVTZ0o/edit?usp=sharing Part 4. \n\nMoreover, we present a table for an overview of the composition of the dataset (submitting version). \n\n| Subsets | Templates|  | | | |\n|---|---|---|---|---|---|\n| **Symbolic**        | Uniform template of variable name “V_number” |                |                        |                            |                     |\n| **Medium and Hard** | **Configuration** | **Crazy Zootopia (80%)** | **Teachers and School (10%)** |                            | **Movie Awards (10%)** |\n|| Forward (50%)| 40%|5%| 5%||\n|| Reverse (50%)|40%|5%|5%||\n\nThe amount of variation permitted in each of our templates is different, richest for \"Crazy Zootopia\" and slightly more limited for the other two templates. We use an un-even partition of templates for the ensemble of our dataset. We are actively expanding the number of available templates for GSM-Infinite, now made 10 (much more than 3 when submitting), and will be released once published. **Please notice that the template proportions are fully reconfigurable, and switching to a different template won’t leads to large difference in performance.** For detailed ablation experiments of templates, please refer to (Appendix F, page 19-20).", "claims_and_evidence": "This paper argues that a long-context reasoning benchmark should offer controllable and scalable complexity, natural noise, and support infinite data generation.The first two points are somewhat realized in GSM-Infinite, but the third point is less clear. The authors have only expanded based on GSM-8K as a seed, and it is not evident whether the proposed method can be applied to various types of mathematical benchmarks.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are solid. I did not find any obvious flaws in the method and experiment sections.", "theoretical_claims": "Not applicable. This paper does not contain any theoretical claims.", "experimental_designs_or_analyses": "The experimental designs based on the GSM-infinite are sound. I also checked the experimental details in the appendix,and I did not notice any obvious flaws.", "relation_to_broader_scientific_literature": "This paper proposes GSM-Infinite, a synthetic dataset for testing reasoning capabilities over long contexts.It serves as a complement to existing mathematical benchmarks and can test a model's reasoning abilities in longer contexts or with introduced noises.", "essential_references_not_discussed": "I did not notice the issue of missing references.", "other_strengths_and_weaknesses": "This paper proposes GSM-Infinite, which synthetically constructs long-context questions based on GSM-8K as seed problems.Neither the dataset itself nor the experiments based on the dataset have significant issues. However, as the authors discussed in Appendix A.4,t here are many synthetic methods for expanding seed questions into long-context reasoning tasks with controllable quality. The pipeline designed by the authors is also rather tailored to GSM-8K, which raises concerns that the novelty and significance of this paper are limited. Therefore, I consider this paper to be at the borderline level.", "other_comments_or_suggestions": "It appears that the construction of GSM-Infinite is based on computational graphs, which can be applied to a broader range of mathematical and reasoning problems. Have you considered constructing multi-source datasets under this method to build“infinite”datasets, which could lead to more universal conclusions?", "questions_for_authors": "Have you manually checked the synthesized questions? Are there any unnatural cases? Have you conducted any statistics on the diversity and quality of the questions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary**: This paper designs a way to controllably generate GSM8K-type problems with varying context length / difficulty / reasoning. The paper then analyzes 18 SoTA LLMs on the generated problems and reports how their performance varies with varying problem complexity, varying types of distractors injected into the problem, varying model sizes/types and also varying inference-time compute (Best-of-N sampling, specifically).

The controllable and scalable benchmark here would be highly valuable for analyses; despite the GSM8K-ness of the benchmark, it makes good progress on top of existing benchmarks; also, the experiments are comprehensive and the reviewers did not find any major flaws.

**Details**: 
Reviewers appreciated the value in how the work here complements other reasoning benchmarks (`GB4k`), isolates long-context reasoning from long-context retrieval (`6Z5g`), and makes data generation scalable (`djUT`). The paper describes its process and findings well (`6KRZ`, `djUT`); the experiments are designed well (`GB4k`); they demonstrate the claims (`djUT`), are comprehensive (`6Z5g `) and the findings are interesting (`djUT`). They did not find any major weaknesses (`fqRq`, `GB4k`) including Reviewer `GB4k` who read the appendix carefully.  I would add that the experimental observations themselves are as one would expect a priori; but there is significant value in establishing them rigorously and comprehensively.

The one concern that stood out as important to me was that the data generated here may be too simple (`fqRq`) or too tailored to GSM8K (`GB4k`) affecting the significance of the dataset. However, I believe that in the context of what exists in literature, this is relatively more complex and is valuable progress. The authors contextualize their work very well in the introduction: existing long-context datasets can either be solved via retrieval as they have easy-to-identify distractors or are not scalable to infinite complexity/infinite data.

Suggestions: Reviewer `6Z5g` had a valuable suggestion about varying the computational graph structures which the authors have responded to. Reviewers `6Z5g`, `GB4k` both requested a qualitative analysis of model errors which the authors responded to. I encourage the authors to present a condensed version of this in the paper. I agree with `6Z5g` regarding improving the description of the computation graph construction. This part was hard to follow. Concretely, it would help to (a) provide more varying examples of two-entity and three-entity variables and (b) be more careful about discussing the intuition (e.g., the argument in the first para of 4.1 Reverse Problems was hard to follow).

Minor note: I would use the word "distractor" or "filler" rather than "noise". Noise to me indicates corrupting the essential values which is different from what is being studied here.
Decision: Accept (poster)