Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective
Authors: Qingchuan Ma
Yuhang Wu
Xiawu Zheng
Rongrong Ji
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Abstract Reasoning
Benchmark
Primary area: deep_learning->large_language_models

Abstract: In this paper, we aim to establish a simple, effective, and theoretically grounded benchmark for rigorously probing abstract reasoning in Large Language Models (LLMs). To achieve this, we first develop a mathematic framework that defines abstract reasoning as the ability to: (i) extract essential patterns independent of surface representations, and (ii) apply consistent rules to these abstract patterns. Based on this framework, we introduce two novel complementary metrics: Γ measures basic reasoning accuracy, while ∆ quantifies a model's reliance on specific symbols rather than underlying patterns - a key indicator of true abstraction versus mere memorization. To implement this measurement, we design a benchmark: systematic symbol remapping in rule-based tasks, which forces models to demonstrate genuine pattern recognition beyond superficial token matching. Extensive LLM evaluations using this benchmark (commercial API models, 7B-70B, multi-agent) reveal:1) critical limitations in non-decimal arithmetic and symbolic reasoning; 2) persistent abstraction gaps despite chain-of-thought prompting; and 3) ∆'s effectiveness in robustly measuring memory dependence by quantifying performance degradation under symbol remapping, particularly highlighting operand-specific memorization. These findings underscore that current LLMs, despite domain-specific strengths, still lack robust abstract reasoning, highlighting key areas for future improvement.

Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective
Qingchuan Ma Yuhang Wu
Xiawu Zheng 12
Rongrong Ji
Abstract
In this paper, we aim to establish a simple, effec- tive, and theoretically grounded benchmark for rigorously probing abstract reasoning Large Language Models (LLMs). To achieve this, we first develop mathematic framework that defines abstract reasoning the ability to: (i) extract essential patterns independent of surface repre- sentations, and (ii) apply consistent rules to these abstract patterns. Based on this framework, we introduce two novel complementary metrics: T measures basic reasoning accuracy, while A quan- tifies model's reliance specific symbols rather than underlying patterns key indicator of true abstraction versus mere memorization. To imple- ment this measurement, we design a benchmark: systematic symbol remapping in rule-based tasks, which forces models to demonstrate genuine pat- temn recognition beyond superficial token match- ing. Extensive L.LM evaluations using this bench- mark (commercial API models, 7B-70B, multi- agent) reveal: 1) critical limitations in non-decimal arithmetic and symbolic reasoning; 2) persistent abstraction gaps despite ohain-of-thought prompt- ing; and 3) A's effectiveness in robustly mea- suring memory dependence by quantifying per- formance degradation under symbol remapping, partieularly highlighting operand-speciff memo- rization. These findings underscore that current LLMs, despite domain-specific strengths, still lack robust abstract reasoning, highlighting key areas for future improvement.
0.8
Score
0.6
0.4
0.2
0
Llana Mamm--33-70pp 3-708
desparok
Qwen2 Math-
Llama-3 Nanotron O
OML Lam
genun-
Qwen2.3 5.78
Ohf Liamas
agenchaa
Llam-33, SB
gemma -hh
Iin dehate
ala-4-ip 1.5h
apermmth 5.999
interdaz 2-77
Marco-o
708
Qwinz
GSM8K
Figure I. Models often excel on GSM8K (blue line) but show sig- nificantly lower performance on our abstract reasoning benchmark (red line). suggesting that domain-specific math tasks may not probe deeper abstract skills.
1. Introduction
Abstract reasoning, cornerstone of human-level intel- ligence (Holyoak & Morrison, 2012: Penn al., 2008; Holyoak, 2012; Chollet, 2019; Bober-Irizar & Banerjee, 2024; Xiong et al., 2024), remains critical yet elusive capa- bility for large language models (LLMs). As philosophically defined, abstract reasoning involves two core processes: ab- straction, the extraction of common features from concrete entities (Murphy, 2004), and reasoning, the derivation of knowledge from existing information (Holyoak & Mor- rison, 2012). Abstraction provides the fundamental units and organizational structure for cognition, while reasoning operates and infers relationships between these abstractions. Their synergy empowers systems to understand the world and solve problems.
'Equal contribution 'Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China. Xiamen University, 361005, P.R. China. PInstitute of Artifi- cial Intelligence, Xiamen University.. Correspondence to: Xiawu Zheng <zhengxiawu@ xmu.edu.cn..
While large language models (LLMs) such as GPT se- ries (Radford et al., 2019; Achiam al., 2023), Llama seris(Touvron et al., 2023a;b; Dubey et al., 2024), and PaLM-2 (Anil et al., 2023) have exhibited remarkable achievements in various benchmarks, high performance does not necessarily imply abstract generalization. Indeed, many benchmark tasks can be tackled via pattern matching
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Benchmarking Abstract Reasoning in LLMs
model's dependence specifie symbols rather than under- lying abstract patterns, thus serving crucial indicator of genuine abstraction versus mere memorization. Based on these rigorous definitions, we further derive three the- orems. This theorem further motivates the core principle for our benchmark design: systematic symbol remapping in rule-based tasks, which compels models to demonstrate gen- uine pattern recognition beyond superficial token matching. Consequently, design our domain- generall benchmark, which, unlike domain-specific, math-centric benchmarks like GSM8K, offers more discerning evaluation through symbolic tasks and abstract rules. This design effectively reduces reliance on memorization and domain knowledge, directly targeting genuine abstract reasoning. We then con- duct extensive evaluations of various LLMs (7B-70B, multi- agent) using this benchmark, revealing key limitations and the diagnostic value of A.
Average Abstract Reasoning Score
0.6
0.4
0.2
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Memory Dependence Score
7B
32B
70B
API
Agents
(1) theoretically grounded framework formalizing abstraction and reasoning, and intro- ducing the and metrics; (2) novel symbolic benchmark with diverse tasks and remapping variants to rigorously test abstract reasoning; and (3) empirical insights from exten- sive LLM evaluations, revealing persistent challenges in non-decimal arithmetic, symbolic transformations, and func- tion inference, even with Chain-of-Thhuueee and multi- -agent methods showing only partial gains. These findings under- score the need for refined benchmarks like ours to guide the development of truly abstractly reasoning Al systems and highlight key areas for future improvement.
Our contributions are threefold:
Figure 2. Memory Dependence Average Score (Small Scale). The corresponding full-size version is available in 6ig.6 of the Appendix.
or memorized heuristics, failing to assess true reasoning.
Numerous benchmarks attempt to assess reasoning, but of- fall short for rigorously evaluating abstract reasoning in LLMs. ARC (Chollet. 2019)'s 2D visual format is fun- damentally misaligned with L.LMs' text-based nature, hin- dering direct evaluation of their reasoning. Benchmarks like GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), while symbolic, risk measuring memoriza- tion of problem-sol solving heuristics rather than true abstract mathematical understanding. Even BIG-Bench Hard (Suz- gun et al., 2023), with its broad task range, lacks focused theoretical grounding for abstraction, potentially allowing solutions via pattern matching without genuine abstraction. These limitations underscore the urgent need for bench- marks that are both LLM-compatible and theoretically de- signed to isolate and measure abstract reasoning, moving beyond surface patterns and memorization (Wu al., 2024; Jiang et al., 2024; Mirzadeh al., 2024).
2. Related Work
Reasoning Benchmarks. Existing abstract reasoning benchmarks are often domain-specific, limiting scope and theoretical interpretability (Chollet, 2019; Barrett et al., 2018; Mirzadeh et al., 2024; Frohberg Binder, 2022; Majumder al., 2024: Li al., 2024a; Yuan et al., 2023). Visual benchmarks like ARC (Chollet, 2019) and PGM (Barrett et al., 2018 8) focus on 2D spatial tasks, misaligned with LLMs' text nature. Mathematical benchmarks (e.g., GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021)), while symbolic, may allow memorization of domain patterns without deep abstract reasoning. BIG-Bench Hard (Suzgun et al., 2023) offers broader tasks but lacks a uni- fied theoretical abstraction framework. Our -onnenl theoretically grounded benchmark with symbolic tasks ad- dresses these limitations for rigorous abstract reasoning evaluation in LLMs.
Therefore, in this paper, we aim to bridge this gap by con- structing mathematically rigorous theoretical framework to formally define abstract reasoning (Chollet, 2019; Mor- ris et al., 2023) to formally define abstract reasoning as a composite process: first, abstraction as extracting essen- tial patterns from concrete inputs through a mapping func- tion, and second, reasoning as applying consistent rules to these abstract patterns to derive conclusions. Based on this framework, we introduce two complementary met- rics to rigorously probe LLMs' abstract reasoning abilities. Specifically, measures the basic accuracy of reasoning, reflecting how well models apply rules, and A quantifies
Multi-Agent Systems and CoT Prompting. Chain-of- thought (CoT) (Wei et al., 2022) and multi-agent frame- works (ReAct (Yao et al., 2023), AutoGen (Wu et al., 2023). LLM Debate (Du et al., 2023)) aim to enhance L.M rea- soning, but evaluations often lack abstract reasoning focus.
2

---

Benchmarking Abstract Reasoning in LLMs
While CoT//ullt-agent methods improve performance on some reasoning tasks, they don't fundamentally address core abstraction gaps our benchmark reveals, especially for symbolic transformations.
2012). These "essential patterns" are crucial for under- standing, categorization, prediction, and reasoning, contrast- ing with superficial details that are intentionally discarded (Holyoak, 2012). In our framework, we represent both con- crete instances and abstract features as strings, suitable for modeling symbolic information processed by LLMs.
Theoretical Perspectives. Broader theoretical underpin- nings, particularly in the domain of neural-symbolic learn- ing and reasoning, are extensively reviewed by (Besold et al., 2021). Theoretical frameworks for abstract reasoning (Barrett al., 2018; Li et al., 2024b; Mitchell et al., 2023;
Abstraction crucial for effective cognition, enabling com- plexity management and generalization in reasoning. Ab- straction, in essence, is information compression and gener- alization from concrete to abstract string representations, ex- tracting core, reasoning-relevan information. This involves mapping a concrete instance to an abstract feature, aiming to distill it into a concise, generalized form that retains essen- tial patterns for reasoning, while reducing complexity and variability. Building upon established perspectives (Murphy, 2004; Goldstone & Barsalou, 1998; Ross & Spalding, 1994), we formally define abstraction as follows:
Chollet, 2019; Morris et al., 2023; Huang & Chang, 2023;
Boix-Adsera et al., 2023; Jiang et al., 2024; Wu et al., 2024) offer insights but are often conceptual or task-specific. Bar- rett et al. (Barrett et al., 2018) explore information theory in visual reasoning; Mitchell et al. (Mitchell et al., 2023) emphasize feature selection in abstraction. However, these often lack direct application to LLM abstract reasoning evaluation. Our work builds on these, offering a unified, grounded framework and benchmark for quantitative LLM evaluation, with metrics measuring invariance and general- ization. Such quantitative evaluation of genuine abstraction erucial, especially as the field explores avenues like neuro- symbolic AI to foster more robust reasoning beyond mere pattern matching (d' Avila Garcez & Lamb, 2020). Fur- thermore, our benchmark's systematic symbol remapping directly probes the generalization of learned patterns and the analogical reasoning capabilities that are increasingly observed in LLMs (Webb et al., 2023).
Definition 3.1 (Abstraction Mapping). Let C be the set of concrete instances, each being individual instance represented as a string of maximum length 1, such that C S Iكو (Σ is finite alphabet). Let A be the set of abstract features, each an abstract feature a E AA also a string with maximum length a S such that A C ΣSa
Abstraction a mapping function CC A.
For example, recognizing a "dog" involves abstraction to generalize across breeds, focusing on essential features like "four-legged" and "mammalian" while disregarding details like "fur color". In our benchmark, a binary addition prob- lem like "010001 10 + 00011111" (concrete instance c) can be abstracted to "binary addition operation" (abstract feature a), focusing on the operation rather than specific operands. Here, I the maximum length of the problem string, and a is the length of "binary addition" (a < I).
3. Theoretical Foundations on Abstract and Reasoning
In this section, we establish a rigorous theoretical frame- work for analyzing and evaluating abstract reasoning in LLMs. While prior research (Holyoak & Morrison, 2012: Penn et al., 2008; Murphy, 2004; Holyoak, 2012; Evans, 2010; Holland, 1986) has explored abstract reasoning from cognitive and philosophical perspectives, we advance the field by providing a formal. mathematically grounded frame- work specifically designed for computational systems. Our framework consists of three complementary components: formal definitions that precisely characterize abstraction and reasoning processes (Section 3.1), theoretical validation that establishes the mathematical soundness of our approach (Section 3.5), and evaluation metrics that enable systematic assessment of abstract reasoning capabilities (Section 3.3).
Reasoning, in cognitive science, is broadly defined de- riving new information from existing knowledge (Holyoak & Morrison, 2012; Evans, 2010). It involves manipulat- ing abstract representations, identifying relationships, and inferring conclusions. This process can be formalized as ap- plying a rule to an abstract feature to produce a conclusion. Building on this, we formalize the reasoning function as:
Definition 3.2 (Reasoning Function). Let R be the set of rules, where each rule r E R is string describing an operation, procedure, or pattern, with maximum length p. such that R. C ySP, Let A be the set of abstract features (Definition 3.1). Let Q be the set of conclusions, each conclusion q E Q also a string with maximum length q, such that Q C Σوe
3.1. Formal Definitions: Abstraction and Reasoning
Abstract reasoning relies on the core processes of abstraction and reasoning. Cognitive science defines abstract reason- ing as identifying and extracting essential, generalizable patterns from complex instances, effectively filtering irrel- evant details to focus on core concepts for understanding and generalization (Murphy, 2004; Holyoak & Morrison,
The reasoning process is formalized by a function Re: A R R+$ Q.
For instance, with the abstract concept "dog" and the rule
3

---

Benchmarking Abstract Reasoning in LLMs
"dogs bark," reasoning infers "This dog is likely to bark" upon encountering new dog. In binary addition, with the abstract feature "binary addition operation" (from A) and a binary addition rule r E R, Re yields a conclusion E Q like "01 100101" from operands "010001 10" and "00011111". Here, q is the maximum conclusion string length.
2024). It enables new knowledge acquisition, adaptation, and generalization beyond direct experience, moving from observation to rule crscovery-crrccall for intelligent behav- ior. This form of reasoning involves inferring rule from examples and then applying it to new instances. Building on cognitive foundations and inductive learning theories (Holland, 1986; Tenenbaum et al., 201 1; Anderson, 1991), we formally define the Rule-Inductive Composite Abstract Reasoning Function:
3.2. Two Types of Composite Abstract Reasoning: Rule-Given and Rule-inductive
Definition 3.4 (Rule-Inductive Composite Abstract Rea- soning Function). When the rule is initially unknown and inferred from examples, the composite abstract reasoning function, H4, incorporates rule induction. Given example set e = (q;) {, and new instance c E C, it is:
Cognitive science identifies rule-based reasoning, or de- ductive reasoning, as applying pre-established rules to de- rive conclusions (Evans, 2010; Johnson-Laird, 1999; Liu et al., 2024b; Wang et al., 2024; Boix-Adsera et al., 2023). Rule-given abstract reasoning focuses on scenarios with ex- plicit or known rules, reflecting our ability to use existing knowledge for inference in new situations comerstone of cognition. This type of reasoning can be formalized as a composite process involving abstraction followed by rule application. Building on these insights and deductive rea- soning research (Evans, 2010; fohnson-Laird, 1999; Laird, 2019), we formally define the Rule-Given Composite Ab- stract Reasoning Function:
Hi(e, = -Re()) f) where i == InferRule(e). (3)
Here, e = [[(ee eq;) eq;)){== is an example set of instance- conclusion pairs (ecj E C, eqj E Q). E C is new concrete instance string. + abstraction mapping, C
Re : A R + Q is reasoning function, i E R is the inferred rule string (max length p), and InferRule(e) - f is the rule inference mechanism.
Consider learning new plant species: we induce rule string from examples of leaf shape, flower color, etc. In our benchmark, inferring operation "#op" from examples like e = { ("2 #op 3", "5"), ... .(7 #op 2", "9") }. Hi first infers rule f InferRule(e) (e.g. "addition"). then for e "5 #op 4", applies Re(f(c), f). yielding q = "9". The system induces the rule from examples, then applies it to new problem.
Definition 3.3 (Rule-Given Composite Abstract Reason- ing Function). When rule r E IR is given, the composite abstract reasoning function, Ho, represents the complete abstract reasoning process. It is formalized as the composi- tion of abstraction mapping f (Definition 3.1) and reasoning function Re (Definition 3.2):
Ho Reo
3.3. Measuring Abstract Reasoning: A Two-Metric Approach
For concrete instance e E C and a given rule r E I,, it is:
Halc, = Re)
(2)
Traditional accuracy metrics, while important for evaluating model performance, are insufficient for assessing genuine abstract reasoning in LLMs (Wu et al., 2024; Jiang et al., 2024; Mirzadeh et al., 2024; Dentella et al., 2023). As dis- cussed in Appendix 1, high accuracy can stem from mem- orizing input-output patterns or exploiting superficial corre- lations-a a Rule-Givee reasoning form based on pre-learned associations. This memorization-basee "reasoning" is brit- tle, failing to generalize when symbolic representations change, even with constant abstract task structure. To evalu- ate true Rule-Inductive abstract reasoning- rule extraction and application independent of specific symbols-metrice must discern memorization from genuine abstraction. To overcome this limitation and rigorously assess genuine ab- stract reasoning in LLMs, we propose two complementary metrics: the Abstract Reasoning Score (T) and the Memory Dependence Score (A). The Abstract Reasoning Score (l') is designed to measure the basic accuracy of a model on abstract reasoning tasks using original symbols, establishing performance baseline.
Here, c E C is a concrete instance string (max length l), r E R is rule string (max length p). f C + A is abstraction mapping, and Re: A x IR + Q is reasoning function. A is the set of abstract features (strings, max length < l), and Q is conclusions (strings, max length q).
Consider recognizing dog breeds: we apply learned "dog- ness" rules to identify even unfamiliar breeds-rules rule-given abstract reasoning in action. In our benchmark, binary addi- tion exemplifies this. For c c "010001 10 + 00011111" and a given binary addition rule r. HG abstracts f(c) to "binary addition operation", then Re(a, r) applies r. yielding q = "01100101". The rule is provided; the task is abstraction and application.
Rule induction, or inductive reasoning, contrasts with rule- given reasoning by learning general rules from specific ex- periences (Holland, 1986; Sloman, 1996; Qiu et al., 2024; Li et al., 2024b; Mirchandani et al., 2023; Merler et al.,
4

---

Benchmarking Abstract Reasoning in LLMs
Definition 3.5 (Abstract Reasoning Score, r). Consider test set T = {(ci, Tú, [1) of N independent tasks. For each task i, e E C is concrete instance string (max length 1), rs E R is rule string (max length p), and E Q is the ground truth conclusion string (max length g).
Smaller A signifies robustness remapping. indicating less token dependence and more abstract, rule-based reasoning. Ideally, a truly abstract reasoning model has A ~ 0, show- ing invariance to symbol remappings.
In summary, I and A offer a comprehensive evaluation of abstract reasoning in LLMs. measures basic reason- ing accuracy under standard conditions, establishing a per- formance baseline. A, as a diagnostic metric, reveals the nature of this accuracy by quantifying performance drop un- der symbol remapping. This distinction helps differentiate accuracy from Rule- Inductive abstraction (low A) versus Rule-Given memorization (high A). This dual-metric ap- proach is crucial for deeper insights into LLMs' abstract reasoning, moving beyond superficial accuracy evaluations.
The Abstract Reasoning Score, I, is calculated as:
N r
N Σ 1 (((.))) = q:).
(4)
where ((((c,)))) - q:| is an indicator function evaluating to I if model fl's predicted conclusion (c,,, r1) matches the ground truth gi (string identity), and 0 otherwise; sum- mation is over all N tasks. H is the model's s composite abstract reasoning functiin.
3.4. Comparison with Existing Abstract Reasoning Benchmarks
A higher I' (0 to 1) indicates greater average accuracy of H on T with original symbols. I measures baseline perfor- mance on abstract reasoning tasks under standard symbolic conditions, showing model performance with familiar rep- resentations, but it does not alone differentiate abstraction from memorization.
Existing benchmarks offer Al reasoning insights, but in- completely assess abstract reasoning per our framework. GSM8K, MATH, BIG-Beneh Hard mainly evaluate Rule- Given Reasoning, susceptible to memorization and lack- ing memory dependence measures. ARC emphasizes Rule- Inductive Reasoming and reduces memorization via novelty, but its visual format is suboptimal for LLMs and neglects Given abilities. Thus, no existing benchmark fully evaluates both Rule-Given and Rule-Inductive reasoning in unified framework, nor offers memory dependence metrics. This critical eappeeallllllly distinguishing abstraction from memorization sieriiaation-hhhligghtts the need for novel benchmarks like ours and A, specifically designed for rigorous, diag- nostic evaluation of abstract reasoning in LLMs by directly addressing both reasoning types and memory dependence.
To assess model dependence on specific symbols and probe Rule-Inductive reasoning, we introduce A. A, the Memory Dependence Score, is designed to quantify a model's depen- dence on specific symbols rather than underlying abstract patterns. The core idea is to disrupt memorization via sym- bol mapping M, which systematically re-labels symbols in concrete instance strings G C of test set T. M preserves abstract task structure while altering surface symbols (e.g.. in binary arithmetic, remap '0', 'I' to 'A', 'B'), forcing reasoning about binary operations, not memorized '0', '1' associations.
Definition 3.6 (Memory Dependence Score, A). Apply symbol mapping M to each & in original test set T = ((c, Ti, W creating symbol-mapped set M (T) {(M()) ri, M(g.)) Rule strings r; remain in the orig- inal symbolic space.
3.5. Theoretical Validity of I and A Metrics in Abstract Reasoning Assessment
While our framework and metrics provide an intuitive ap- proach to evaluating abstract reasoning, we need to estab- lish their theoretical soundness. Just as cognitive science research validates psychological measures through formal analysis, we must demonstrate that our metrics I and A truly capture the intended aspects of abstract reasoning. Here, we develop three foundational theorems that validate our approach.
Evaluate model A on M(T), calculating accuracy I'M, similar to T:
1 N
IM = Σ (((( N
(5)
Our first theorem addresses a fundamental question: Can we trust [ reliable indicator of an LLM'' basic reasoning capabilities? ? This validation is essential before can build upon it to assess more complex reasoning patterns.
The Memory Dependence Score, A, is the difference be- tween I and M
A r - T-TM-
(6)
Theorem 3.7 (Validity of I for Rule-Given Potential). Let H be an LLM approximatiigg the abstract reasoning function H = Re o f. For a sufficiently high threshold 7 E [0, 1]:
quantifies performance degradation under remapping. Larger A indicates significant performance drop, suggest- ing high dependence on original symbols and memorization.
((f((r))) = a (c,r.)) E TT
(7)
5

---

Benchmarking Abstract Reasoning in LLMs
where T is the test set of concrete instances, rules, and expected conclusions.
test for genuine abstraction, moving beyond mere token memorization. Tasks are theoretically aligned with the def- initions of abstraction (f) and reasoning (Re) established in Section 3, and are categorized (BC-SR) for structured analysis. This tiered category system (BC-SR) also ensures scalability across varying levels of cognitive complexity, from basic arithmetic to function inference. Finally, objec- tive diagnosis of abstract reasoning and memory dependence is achieved through quantitative metrics, I and A and supported by automated analysis.
This theorem establishes that achieving a high I' score indi- cates strong Rule- Maile- reasoning potential in the model.
With the foundation of I's validity established, we turn to more nuanced question: How can we ensure that high per- formance truly reflects abstract understanding rather than memorization? This leads us to our second theorem, which validates A measure of genuine rule-inductive reason- ing.
4.2. Task Categories
Theorem 3.8 (Validity of A for Rule-Inductive Abstraction). For sufficiently small 8 [0, 1] and high y E [0, 1]:
Our benchmark tasks are designed with tiered complexity to comprchensively evaluate abstract reasoning. Basic Com- putation (BC) tasks assess fundamental arithmetic skills in the decimal system, serving as a bascline for computational abilities. Extended Calculation (EC) tasks evaluate the abil- ity to perform diverse computations beyond basic arithmetic, testing the breadth of computational skills. Number Base Reasoning (NBR) tasks rigorously test the generalization of arithmetic reasoning beyond decimal, forcing abstrac- tion of underlying arithmetic principles. Math Application (MA) tasks evaluate multi-step mathematical word prob- lem solving, assessing higher-level reasoning in applied settings, adopting GSM8K style problems for comparison. Symbolic Math Abstraction (SMA) tasks probe inductive mathematical reasoning and abstracting symbolic functions from numerical data, testing pattern discovery. Symbolic Reasoning (SR) tasks directly assess abstract rule applica- tion and symbolic manipulation, testing rule identification and application with abstract symbols, independent of do- main knowledge.
A < 88122
This theorem demonstrates that low memory dependence combined with high accuracy indicates true Rule-Inductive abstraction capabilities.
Having established the validity of both metrics individu- ally, we naturally arrive at the question of how to interpret them in combination. Our final theorem provides a formal framework for this unified interpretation.
Theorem 3.9 (Score Range Interpretation).
F(T,)) = un T + w2(11 A)
(9)
where W1, w2 2 0 and w1 + w2 =
This theorem establishes a continuous mapping F [0, 1] x [0, + [0, 1] that provides a valid measure of abstract reasoning ability by combining both metrics.
Detailed task specifications and examples are provided in the appendix.
These theorems provide a rigorous mathematical foundation for our evaluation framework, demonstrating that I' and A effectively capture different yet complementary aspects of abstract reasoning ability in LLMs. The proofs for all theorems are presented in Appendix A.2.
5. Experimental Results and Analysis
In this study, we assessed the abstract reasoning abilities of Large Language Models (LLMs) across different scales and types. Our evaluation encompassed 7B-scale and 70B- scale open-source models, APl-accessible models, and agent frameworks (agentchat(auutogen)) react, IIm debate), the lat- ter leveraging gpt 4opt-4o-om -ommiii For open-source and API models, we employed greedy decoding for inference, while agent frameworks were used with default settings. To evalu- ate performance, we utilized two prompting strategies for each model type: Direct Prompting (DP) and Chain-of- Thought (CoT), primarily zero-shot CoT, with the excep- tion of Math Application (MA) tasks under CoT, where 8-shot examples based GSMSK-lik -ike data were incorpo- rated. All experiments were conducted on local machines equipped with 8 NVIDIA GPUs, including A800 and 3090 models. For consistent and reliable evaluation of model
4. Benchmark Design
We designed a novel symbolic task benchmark to rigorously evaluate abstract reasoning in LLMs, operationalizing our theoretical framework and addressing limitations of existing evaluations. This benchmark provides direct and robust assessment of abstract reasoning capabilities.
4.1. Benchmark Design Principles: Theoretical Grounding
Our benchmark design, theoretically grounded in Section 3, prioritizes effective LLM evaluation of abstract reasoning. We use -i-dimensional text input for relevance to LLMs. Systematic symbol remappings are employed to rigorously
6

---

Benchmarking Abstract Reasoning in LLMs
Table Abstract Reasoning Performance (T Scores) of Represen-
tative LLMs across Different Scales and Types.
Score (L)
0.8
0.6
MODEL
AVERAGE SCORE
0.4
7B-SCALF MODELS
0.2
GLM-4-8 CLLM-4-9B-CE 9B-AATT
0.17
0
CHAT
BC
EC
NBR
MA
SMA
SR
Avg
GEMMA-2-9H-IT
0.18
REMMA-2-9B-IT COT
0.19
7B Normal
7B CoT
708 Normal
70B CoT
LLAMA-3.1-8B I.TSRRUT
0.17
Agenis Normal
LLAMA-3. I-SSSRUTCCCTT COT
0.21
0.19
QWEN2.5.7D. 7B IISTTRUCTC COT
0.34
32B-SCALE MODELS
Figure 3. Abstract Reasoning Performance across Tasks, Model Types, and Prompting (Normal/CoT).
QWQ-32B-PRIVIEW QWQ-32B-BBBB PREVIEW COT
0.22
0.50
70B-SCALE MODELS
MemDep Score (A)
11.MM--3.--0R- INSTRUCT
-
0.22
DAM-A-33--7B--INSS
COT
0.43
0.4
QWEN2.5-72B-o INSTRUCT
QWEN2.5-72B INSTRUCT COT
0.47
API-BASED MODELS
0.2
GPT-40-MINI GPT-4O-MINI COT
0.40
0.43
GEMINI-2.0 O-F-LSSI--EXP
0.29
GEMINI-2.0CM GEMINI-2.--FLASI--- O-FLASH-EEXP COT
0.52
GEMINI-2.00 2.0-FLAHH THINKING-ENY O--LLAS-TTHINKINGGGG
0.39
7B-Scale
70B- McB-Scale
API-Bused
2.0. FLASH THIHKINININNG.AA
0.54
DEEPSEEK V3
0.48
MemDep.op
MemDep.op (CoT) 0 MemDep.
0.41
MemDep num (CoT) MemDep all
MemDep all (CoT)
FRAMEWORKS
AGENTCHAT(AITOGEN)
0.60
LLM DEBATE
0.35
Figure 4. Memory Dependence Score (A) across Model Types and Prompting Strategies. Operand memory dependence is consistently higher.
outputs across all tasks, we employed gpt-4o 1ott-4o-mnni to parse responses and determine answer correctness. Based on this comprehensive evaluation framework, our experi- ments revealed several critical insights into current LLM capabilities and limitations.
task categories. Figure presents a detailed comparison of Abstract Reasoning Score (I') across model types and tasks. On decimal-based tasks (Basic Computation (BC), Extended Calculation (EC)), API models achieve average 1 exceeding 0.5, while 7B/70B models reach over 0.4 (Table 3 for detailed scores). However, Number Base Reasoning (NBR) performance collapses across all model scales, with average below 0.1, highlighting a significant abstraction gap numerical generalization.
Through systematic evaluation focusing rule-inductive reasoning and generalization beyond memorized patterns, our analysis uncovered several fundamental limitations in current LLMs: (I) widespread failure in non-decimal arith- metic reasoning, with even 70B-scale models showing near- zero performance on number base tasks; (2) strong depen- dence on specific operand symbols rather than abstract pat- terns, as quantified by our Memory Dependence Score A; and (3) complex trade-off in Chain-of- Chain-of-Thought prompting, where improved task performance often comes at the cost of increased memory dependence. These findings suggest fundamental limitations in LLMs' ability to perform gen- uine abstract reasoning, particularly when faced with novel symbolic representations.
Math Application (MA) tasks show substantial gains with Chain-of-Thought (CoT) prompting, particularly for 7B models, suggesting CoT's effectiveness in guiding multi- step, familiar arithmetic problems. In contrast, Symbolic Reasoning (SR) and Symbolic Math Abstraction (SMA) tasks remain challenging, with average generally below 0.3 even for larger models, indicating persistent difficulty in abstract symbolic manipulation and function inference.
The Memory Dependence Score A provides crucial in- sights into the nature of model capabilities. Analy- sis reveals consistently higher dependence on operand symbols (MemDep num) compared to operator symbols (MemDep op) across models and tasks. For instance,
5.1. Quantitative Results and Analysis
Our evaluation encompasses performance metrics across different model scales (7B-70B), prompting strategies, and

---

Benchmarking Abstract Reasoning in LLMs
rule induction for novel symbolie systems, revealing cognitive capabilities currently lacking in LLMs.
Table Memory Dependence Analysis (A Scores) of Representa-
tive LLMs: Measuring Abstraction vs. Memorization.
Chain-of- Shain-of-Thought prompting exhibits nuanced effects across different task categories. While it reduces arithmetic errors in MA and EC tasks, shows limited benefit for NBR, SMA, or SR tasks. This pattern suggests that CoT primarily aids in structuring procedural steps within familiar domains rather than enhancing fundamental abstraction capabilities. Moreover, the concurrent rise in memory dependence (A) with CoT in some cases indicates a potential trade-off: pro- cedural guidance might reinforce token-specing reasoning at the expense of genuine abstraction.
MODEL
NUM
ALL
7B-SCALE MODELS
GLM GLM-4-98 CHAT
0.14
0.26
0.27
GIM-4-9R-CHAT COT
0.30
0.34
GEMMA-2-98-IT
0.10
0.37
0.37
GEMMA-2-9B-IT COT
0.33
0.36
0.40
LLAMA-3.1-88 8---NSTRUCTT
0.11
0.35
0.35
LLAMA-3.1 3.1-88...BB 8B. INSTRUCT COT
0.14
0.23
0.23
OWEN2.5-7B-INSTRUEE
0.10
0.36
0.37
QWEN2,5-7B INSTRUCT COT
0.23
0.34
0.40
32B-SCALE MODELS
QWQ-32B-PAEVEE
0.23
0.21
QWQ-32B-PREVDDE PREVIEW COT
0.31
0.50
0.58
MA tasks, despite their word problem format, proved rela- tively easier across models, largely attributable their re- liance on familiar decimal arithmetic and extensively trained problem-solving patterns. This observation further supports our finding that models excel in domains with familiar sym- bolic representations but struggle with novel abstract pat- terns.
70B-SCALE MODELS
TLAMA-3.3-70R. --NT--NSUUUCTT
0.11
0.42
0.43
LLAMA-3. -LAMA-3.3-700--1 3..--000-NNSTUUCTTT COT
0.17
0.37
0.43
OWEN2.5-77B-INSTRUUS
0.42
QWEN2,5-72L 72B INSTRUCT COT
0.12
0.41
0.47
API-BASED MODELS
GPT-4O-MINI GPT-4O-MINI COT
0.11
0.30
0.35
0.20
0.47
0.44
Agent frameworks, despite showing improved accuracy scores, exhibited persistent token memorization patterns in SMA and SR tasks. This suggests that even sophisticated multi-agent architectures fail to overcome fundamental ab- straction limitations, instead potentially reinforcing superfi- cial pattern matching through their interaction protocols.
GEMINI-2. O-FLASH-EEXP 0.09
0.44
GEMINI-2.0 D-ELASH-EXX COT
0.15
0.35
0.41
GEMINI-2 2.0NK-2.0-FE. FLASH-THINKINGG
0.10
0.48
0.51
GEMINI-2 2.0 THINKINNG EXP
0.08
0.33
0.41
DEEPSBEK V3
0.08
0.37
0.41
0.40
0.47
AGENTS FRAMEWORKS
AGENTCHAT(AITOGEN)
0.25
0.50
0.56
REACT
0.70
LLM DEBATE
0.21
0.40
0.41
5.3. Deeper Analysis: Training Data Impact and Human Baseline
To further probe the nature of LLM abstract reasoning, we conducted supplementary experiments investigating the im- pact of training data and establishing a human performance baseline, Detailed experimental setups is provided in Ap- pendix A.
exhibits MemDep num = 0.42 vs. MemDep = 0.11, indicating stronger reliance on spe- cific operand symbols. CoT prompting sometimes increases A, as observed in glm-4-9b-chat where ANIM rises from 0.26 to 0.30 with CoT. Even agent frameworks show high memory dependence, with react reaching MemDep_an of 0.70, suggesting persistent token-specific reasoning despite architectural sophistication.
Llama-3.2 3.3-
First, we fine-tuned Llama-3.1 --s--unst datasets with and without our systematic symbol remapping. Fine-tuning on data containing remapped symbols significantly im- proved performance on tasks with the same remapping structure (e.g., accuracy on the fixed_len_chat.bll "attaaaaaibiddattsee" task increased from 13% to 60% for direct prompting after fine-tuning remapped data). However, this im- provement showed limited generalization; performance on tasks with different, unseen remapping structures (e.g.. "fixed Lexed_len_de len den_chat..o str "tr_dattett") did not show similar uplift and, in some cases, even degraded slightly (see Table 4). This suggests that the model tended to memorize specific mapping patterns rather than acquiring a generalizable ab- stract rule-application capability.
5.2. Pattern Analysis and Failure Modes
Our analysis reveals clear task difficulty hierarchy: SMA(hardest) - NBR - SR + EC, BC - MA (casiest), reflecting fundamental limitations in LLMs' abstract reason- ing capabilities. This is particularly evident NBR tasks, where low I' scores across all model scales demonstrate a critical failure to generalize arithmetic abstraction (ff) be- yond decimal representations, often triggering catastrophic failures random outputs, especially in smaller models. Similarly, SMA tasks frequently yielded apparently random responses, pointing to a fundamental weakness in inductive function inference- the ability to abstract symbolic rules (Re) from numerical examples. These patterns signify not merely performance limitations, but fundamental deficits in
Second, we benchmarked human performance using un- dergraduate students with a computer science background on a subset of these challenging tasks. Humans demon- strated robust abstract reasoning, achieving serfect
8

---

Benchmarking Abstract Reasoning in LLMs
accuracy (97%) on the remapped bitwise operation task ("fixed len chat bit dataset") and strong performance (87%) on non-decimal arithmetic (e.g.. aadd_bae33.mm wtw.rraaaaaaaaao www_datate"" significantly outperforming the LLMs (Table 4). Even on the more complex remapped string operation task ("fixed_enncchatt len_chat_str dataset"), where LLM performance was particularly low, humans achieved 7% accuracy, highlight- ing a substantial gap.
found deficit in abstract symbolic reasoning, hindered by significant memory dependence and limited generalization, even with advanced techniques. The benchmark dataset, gen- eration scripts, and evaluation code proposed in this paper will be publiely available at https: / / //////tttuu.. thub. com/ aww.ractsttatccceee otstttttllaabbtraatatt reason benchmark.
Acknowledgements
These findings underscore that while specific training can hance performance on familiar remapped structures, current LLMs still struggle with genuine generalization in abstract symbolic reasoning and fall considerably short of human capabilities in flexibly applying abstract rules to novel rep- resentations.
This work was supported by the National Science Fund for Distinguished Young Scholars (No.62025603), the Na- tional Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. U23A20383, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), ) and the Natural Science Foundation of Fujian Province of China (No. 2021J06003, No. No.2022106001).
5.4. Implications and Future Directions
The Memory Dependence Score A serves as a critical diag- nostic tool, revealing how LLMs rely more heavily on mem- orizing numerical tokens (MemDep_mut num > MemDep_opl than learning abstract rules. This operand-specific mem- orization fundamentally limits models' ability to general- ize across different symbolic representations. As Figure 6 demonstrates, higher A directly correlates with reduced generalization capability.
Impact Statement
This work introduces novel metrics, I' (Abstract Reasoning Score) and A (Memory Dependence Score), to rigorously benchmark abstract reasoning in Large Language Models (LLMss). The primary positive impact lies in fostering the de- velopment of more robust and generalizable AL. By provid- ing more nuanced evaluation framework that disentangles genuine reasoning from memorization, our benchmark (T) and diagnostic tool (A) can guide research towards LLMs capable of deeper understanding and more reliable perfor- mance in novel, complex scenarios. This could accelerate progress in Al safety and alignment by promoting models that operate underlying principles rather than superficial correlations, potentially leading to more predictable and interpretable systems. Improved abstract reasoning capabil- ities are crucial for advancing Al applications in science, education, and critical mroblem-slvvin domains where true generalization is paramount.
These findings indicate that LLMs' abstract reasoning limita- tions stem not from scale issues, but from their fundamental reliance on memorized patterss rather than representation- invariant rules. Advanced techniques like multi-agent frame- works and CoT prompting improve performance metrics but fail to address this core abstraction deficit.
We propose three critical research directions: (1) Symbolic data augmentation through systematic permutation to reduce token memorization,
(2) Development of functional embed- ding spaces to promote abstraction beyond lexical forms, and (3) Extension of benchmarks to evaluate generalization across physical and causal reasoning domains.
Potential negative impacts, common to advancements in AI evaluation, include the risk of "benchmark hacking", where models might be narrowly optimized for T and A without commensurate gains in holistic cognitive abilities, Further- more, while our work aims to improve LLM robustness, the development of more capable AI systems inherently carries dual-use concerns and necessitates ongoing ethical scrutiny regarding their deployment and societal consequences. The insights gained from these metries, if not carefully contex- tualized, could also be misinterpreted, leading to an over- estimation or underestimation of AI capabilities in specific reasoning facets. We encourage the community to use these tools responsibly, as part of a broader suite of evaluations, to cultivate AI that is not only more capable but also more aligned with human values and societal benefit.
The A metric provides a quantitative measure for track- ing progress toward true symbolic generalization, essential for developing AI systems with genuine abstract reasoning capabilities rather than mere pattern matching.
6. Conclusion
We tackled the crucial challenge of rigorously evaluating abstract reasoning in LLMs through a theoretically robust framework. By defining abstract reasoning as the interplay of abstraction and reasoning, we derived validity for our met- rics, I and A, and designed symbol remapping benchmark to compel genuine generalization. Extensive evaluations us- ing this benchmark exposed a critical limitation: current LLMs, despite strengths in familiar domains, exhibit a pro-
9

---

Benchmarking Abstract Reasoning in LLMs
References
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., et al. The llama 3 herd of models. arXiv preprint 2rXiv2240..21783, 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08777 2023.
Evans, S. B. Thinking twice: Two minds in one brain. Oxford University Press, 2010.
Anderson, J. R. Is human cognition adaptive? Behavioral and brain sciences, 14(3):77--455, 1991.
Frohberg, J. and Binder, F. CRASS: A novel data set and benchmark to test counterfactual reasoning of large language models. In Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isa- hara, H., Maegaard, B., Mariani, J., Mazo, H., Odijk, J., and Piperidis, S. (eds.), Proceedings of the Thir- teenth Language Resources and Evaluation Conference, pp. 2240,, Marseille, France, June 2022. Euro- pean Language Resources Association. URL https: // /acclanthology org/ www.ee--c......
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
Barrett, D., Hill, F,, Santoro, A., Morcos, A., and Lillicrap,
T. Measuring abstract reasoning in neural networks. In International conference machine learning, pp. 511- 520. PMLR, 2018.
Besold, T. R., Bader, S., Bowman, H., Domingos, P., Hit- zler, P. Kühhberger, K.-U., Lamb, L. C., Lima, P. M. V., de Penning, L., Pinkas, G., et al. Neural-symbolic learn- ing and reasoning: A survey and interpretation 1. In Newro-synbolic artificial intelligence: The state of the art, pp. 1-51. IOS press, 2021.
GLM, T., Zeng, A., Xu, B., Wang. B., Zhang, C., Yin, D., Zhang, D., Rojas, D., Feng, G., Zhao, H., et al. Chatgim: A family of large language models from glm- 130b to glm-4 all tools. arXiv preprint arXiv:2406. 12793, 2024.
Goldstone, R. L. and Barsalou, L. W. Reuniting perception and conception. Cognition, 65(2-3):231-262 1998.
Bober-Irizar, M. and Banerjee, S. Neural networks for abstraction and reasoning: Towards broad generalization in machines. arXiv preprint arXiv:2402.0300 2024.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https: http:::://ppeeeee /pppreeeeeeeeeeee/e///e net forum? i dd= 7ByttttttCSCC
Boix-Adsera, E.,, Saremi, O., Abbe, E., Bengio, S., Littwin, E., and Susskind, J. When can transformers reason with abstract symbols? arXiv preprint 1rXiv:2310.09753, 2023.
Cai. Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., et al. Internim2 technical report. arXiv preprint arXiv:2403. 17297, 2024.
Holland, J. Induction; Processes of inference, learning, and
discovery, volume 292. MIT Press, 1986.
Holyoak, K. J. Analogy and relational reasoning. The
Chollet, F. On the measure of intelligence. arXiv preprint 191.0:1911.01547, 2019.
Oxford handbook of thinking and reasoning, pp. 259.259. 2012.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. 14168, 2021.
Holyoak, K. J. and Morrison, R. G. The Oxford handbook of thinking and reasoning. Oxford University Press, 2012 2.
Huang, J. and Chang, K. C.-C. Towards reasoning in large language models: A survey. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 1049-1065, Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: 10. 10.186533112023..fll fing.nigs..iiiinn--a 67. URL. https: a a aeceannhhllco ogy. 2023. Findings-ache 67/.
d' Avila Garcez, A. and Lamb, L. C. Neurosymbolic ai: the 3rd wave. arXiv e-prints, pp. arXiv-2012, 2020.
Dentella, V., Murphy, E., Marcus, G., and Leivada, E. Test- ing performance on less frequent aspects of language re- veals insensitivity to underlying meaning. arXiv preprint WrXiv:2302.12315 2023.
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mor- datch, I. Improving factuality and reasoning in lan- guage models through multiagent debate. arXiv preprint arXiv:2305. /4325, 2023.
Jiang, B., Xie, Y., Hao, Z., Wang, X., Mallick, T., Su, W. J., Taylor, C. J., and Roth, D. A peek into token bias: Large language models are not yet genuine reasoners. arXiv preprint arXiv:2406. 1050, 2024.
10

---

Benchmarking Abstract Reasoning in LLMs
Johnson-Laird, P. N. Deductive reasoning. Annual review of psychology, 50(1):109-133, 1999.
Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., and Farajtabar, M. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05223) 2024
Laird, E. The Soar cognitive architecture. MIT press, 2019.
Mitchell, M., Palmarini, A. B., and Moskvichev, A. K. Comparing humans, GPT-4, and GPT-4v abstrac- tion and reasoning tasks. In AAAI 2024 Workshop on "Are Large Language Models Simply Causal Parrots?", 2023. URL https:: wwwp//pppppppeeeeeeeepeee net forum? id=3rGT50k2pC.
Li, C., Yuan, Z., Yuan, H., Dong, G., Lu, K., Wu, J., Tan, C., Wang, X., and Zhou, C. MuggleMath: Assessing the impact of query and response augmentation on math reasoning. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 10230-10258, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10. (11863331112222222 1 URL https:// ac aclantho . wwtp://ggg///o/ aal-longgllnnngnogoooo long. 551/.
Morris, M. R., Sohh-Dickstein, J., Fiedel, N., Warkentin, T., Dafoe, A., Faust, A., Farabet, C., and Legg, S. Levels of agi: Operationalizing progress on the path to agi. arXiv preprint arXiv:2311.0246; 2023.
Murphy, G. The big book of concepts. MIT press, 2004.
Li, W.-D., Hu, K., Larsen, C., Wu, Y., Alford, S., Woo, C., Dunn, S. M., Tang, H., Naim, M., Nguyen, D., et al. Com- bining induction and transduction for abstract reasoning. arXiv preprint ar arXiv:2411.02277 2024b.
Penn, D. C., Holyoak, K. J., and Povinelli, D. Darwin's mistake: Explaining the discontinuity between human and nonhuman minds. Behavioral and brain sciences, 31 12):109-130, 2008.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C.. Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Qiu, L., Jiang, I.., Lu, X., Sclar, M., Pyatkin, V., Bhaga- vatula, C., Wang, B., Kim, Y., Choi, Y., Dziri, N., and Ren, X. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypoth- esis refinement. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / //openreview.co net www.m/oddudtodoooooooooooo
Liu, T., Xu, W., Huang, W., Wang, X., Wang, J., Yang, H., and Li, J. Lfgic-of-thoughr Injecting logic into con- texts for full reasoning in large language models. arXiv preprint ar arXiv:2409. 17539, 2024b.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Majumder, B. P., Surana, H., Agarwal, D., Mishra, B. D., Meena, A., Prakhar, A., Vora, T., Khot, T., Sabharwal, A., and Clark, P. Discoverybench: Towards data-driven discovery with large language models. arXiv preprint arXiv:2407.01725, 2024.
Ross, B. H. and Spalding, T. L. Concepts and categories. In Thinking and problem solving, pp. 119 119-148. Elsevier, 1994.
Sloman, S. A. The empirical case for two systems of rea- soning. Psychological bulletin, 119(1):3, 1996.
Merler, M., Haitsiukevich, K., Dainese, N., and Martti- nen, P. In-context symbolic regression: Leveraging large language models for function discovery. Fu, X. and Fleisig, E. (eds.), Proceedings of the 62nd An- nual Meeting of the Association for Computational Lin- guistics (Volume 4: Student Research Workshop), pp- 427-444, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/11/0244 acl sr..9. URL https:: http:::s/acaaatthooltoo org/
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., and Wei, J. Challenging BIG-bench tasks and whether chain-of-thoughe can solve them. In Rogers, A., Boyd- Graber, J., and Okazaki, N. (eds.), Findings of the As- sociation for Computational Linguistics: ACL 2023, pp. 13003-13051 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 00.18653/v1/023: findings-acl.824 URI. https: ac anthology. o org erg/20233 http/222..ffinnnnge.... findings acl. 824/
2024. strelsssw.999
Mirchandani, S., Xia, F., Florence, P., brian ichter, Driess, D., Arenas, M. G., Rao, K., Sadigh, D., and Zeng, A. Large language models as general pattern ma- chines. In 7th Annual Conference on Robot Learning, 2023. URL ptttps://ppeereeeee wwweeeeiwwwweeeeeeeeeeeeeeeeeeee net/forum? dd=R 8MSyE.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.. Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. / 1805, 2023.

---

Benchmarking Abstract Reasoning in LLMs
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, I.., Mesnard, TT,, Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.000 18, 2024.
Wu, Z., Qiu, L.,, Ross, A., Akyürek, E., Chen, B., Wang. B., Kim, N., Andreas, J., and Kim, Y. Reasoning reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Con- ference of the North American Chapter of the Associa- tion for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers), pp. 1819-1862, Mexico City, Mexico, June 2024. Association for Compu- tational Linguistics. doi: 10. 10.18653/ 102. URL https::/t/// http//alaantttt www.hhoool.................. wow.:lllooooooo//2/2
Team, Q. Qwq-32b: Embracing the power of reinforce- ment learning, March 2025. URL https: qwenlm. github. i htwll//////////.oo//ooooooooo/// hoplllgggggg200/ ht////wq33///// 32b/.
Tenenbaum, J. B., Kemp, C., Griffiths, T. L., and Goodman, N. D. How to grow a mind: Statistics, structure, and abstraction. science, 33166022)1279--2225 2011.
naacl 1ong..1on.111 102/.
Xiong, K., Ding. X., Liu, T., Qin, B., Xu, D., Yang. Q., Liu, H., and Cao, Y. Meaningful learning: Enhancing abstract reasoning in large language models via generic fact guid- ance. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. URL https: / //openreview net/ formmfrrmmmmmm wwteirddd-TITGOTTT..
Toshniwal, S., Du, W., Moshkov, I., Kisacanin, B., Ayrapetyan, A., and Gitman, I. Openmathinstruct-2: Accelerating ai for math with massive open-source in- struction data. arXiv preprint arXiv:241000166, 2024.
Touvron, H., Lavril. T., Izacard, G., Martinet, X., Lachaux, M.-A,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.1 02000...3977 13971, 2023a.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng. B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J.. Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang. K., Yu, L.,, Li, M., Xue, M., Zhang. P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y.. Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report. arXiv preprint
Touvron, H., Martin, L.., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
2024.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The Eleventh In- ternational Conference Learning Representations, 2023. URL https: /ttps:t/////peeeeieeee net forum? <d=WE_vluYUL-
Wang, S., Wei, Z., Choi, Y., and Ren, X. Can LLMs reason with rules? logic scaffolding for stress-testing and im- proving LLMs. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 7523-7543, Bangkok, Thailand, August 2024. Association for Computational Linguis- URL https:
Young, A., Chen, B.,, Li, C.,, Huang, C., Zhang, G., Zhang, G., Wang. G., Li, H., Zhu, J., Chen, J., et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.0440 2024.
tics. doi: 10. 10.18655/w11/0022.clllloggg //aclanthologeeeeeeeee org/ acl- acl-long 406/.
Webb, T., Holyoak, K. J., and Lu, H. Emergent analogi- cal reasoning in large language models. Nature Human Behaviour, 70):1122-1144,, 2023.
Yuan, Z., Yuan, H., Li, C., Dong, Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning math- ematical reasoning with large language models. arXiv preprint 2rXiiv2308.0882 2023.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., brian ichter, Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https::/p/// http://ppneeeeeeeeeeeeeeee net/ Forum?ice Beeiddvvjjv0 ViQ_MeSBB..
Zhao, Y., Yin, H., Zeng, B., Wang, H., Shi, T., Lyu, C., Wang, L., Luo, W., and Zhang, K. Marco-ol: Towards open reasoning models for open-ended solutions. arXiv preprint arXiv:2411. 14405, 2024.
Wu, Q. Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E., Li, B., Jiang, L., Zhang, X., and Wang, C. Autogen: Enabling next-gen IIm applications via multi-agent conversation framework, arXiv preprint arXiv:2308.00 200.:2308.08155, 2023.
12

---

Benchmarking Abstract Reasoning in LLMs
A. Appendix
In this appendix, we present comprehensive performance tables of our experimental results, including I and metrics across various models and tasks. In addition to these detailed tables, we provide further clarifications on symbol mapping protocol to ensure a complete understanding of our benchmark and evaluation process.
A.1. Abstract Reasoning in Machine Learning Context
From a statistical learning perspective, the composite abstract reasoning function H o class in machine learning. Here, the abstraction mapping f acts as a feature extractor, transforming concrete inputs into abstract representations, and the reasoning function Re serves decision rule, operating these abstract features to produce conclusions. In the context of Large Language Models (LLMs), this perspective provides valuable insights. The abstraction mapping f can be loosely associated with the embedding and encoding layers of an LLM, which process input tokens and extract relevant features. The reasoning function Re, on the other hand, aligns with the decoder and prediction components, which utilize the extracted features to generate output sequences.
f can be viewed as hypothesis
However, applying this framework to machine learning. especially in the context of evaluating abstract reasoning in LLMs, introduces unique challenges that distinguish it from traditional machine learning approaches:
1. Symbolic Invariance: Traditional machine learning data augmentation techniques, like image rotation or cropping, aim to improve robustness to variations in input data. For abstract reasoning, however, we require more nuanced form of invariance: symbolic invariance. This means that a truly abstractly reasoning model should be invariant to symbolic transformations that alter the surface form of inputs (e.g., symbol remapping) but preserve the underlying abstract structure and rules. As illustrated in Figure 5, remapping digits or operators should not fundamentally impair the model's ability perform the reasoning task if has genuinely grasped the abstract principles. This is in stark contrast to memorization- based approaches that rely on specific token identities.
2. Composite Learning: In many traditional machine learning tasks, feature extraction and decision makinn might be implicitly learned or treated as separate stages. In abstract reasoning, however, the emphasis on learning a composite process H- Re ʃ. This means that the model must learn seamlessly integrate the abstraction mapping f and the reasoning function Re. It's not sufficient to just have good feature extractor good rule applier in isolation; the model must learn to systematically unify these components to effectively perform abstract reasoning across different rules and tasks. This composite nature requires more holistic learning approach compared to simply optimizing individual modules.
3. Information Compression through Abstraction: Beyond standard generalization metrics like accuracy, abstract reasoning highlights the critical role of information compression through abstraction. A hallmark of abstract reasoning is the ability to distill complex, concrete instances into concise, abstract features. This compression is not merely about reducing dimensionality; it's about selectively discarding irrelevant surface details while preserving the essential, geasoning-relevann structure. An effective abstract reasoning system should operate primarily on these compressed abstract representations, rather than directly on the high-dimensional raw input space. The efficiency and generalizability of abstract reasoning are intrinsically linked this ability to achieve meaningful information compression.
4. Rule Availability during Training (Rule-Given vs. Rule- Inle-Inductive): The nature of abstract reasoning tasks, in relation to an LLM's s training data, significantly impacts the difficulty and the type of reasoning required. Ruve-given abstract reasoning tasks are those where the underlying rules or patterns are likely to have been encountered, implicitly or explicitly, during the LLM's extensive pre- pre-training. In such cases, the model can potentially leverage its vast learned knowledge and memory to perform the task, effectively recalling and applying pre-existing associations. For example, tasks involving decimal arithmetic, common sense reasoning based on everyday knowledge, or applying frequently seen symbolic transformations might fall into this category. The model's performance could be enhanced by its ability to access and utilize relevant information acquired during training.
Conversely, rule- inleinnucttiv abstract reasoning tasks present much greater challenge. These tasks involve novel rules or patterns that are unlikely to have been directly or frequently encountered in the training data. To succeed, the LLM must go beyond simple recall and pattern matching; it must genuinely induce the underlying rule from the provided task description or examples and then apply this newly learned rule to solve the problem. Tasks involving arithmetic in unfamiliar number bases, inferring novel symbolic operations from a few examples, or applying abstract rules defined
13

---

Benchmarking Abstract Reasoning in LLMs
using novel symbols characteristic of rule-inductive reasoning. These tasks demand higher level of abstraction, requiring the model to actively learn and generalize rules on-the-fly, rather than relying on pre-existing knowledge. Therefore, rule-inductive tasks crucial for rigorously evaluating the true abstract reasoning capacity of LLMs, as they minimize reliance on memorization and maximize the need for genuine, flexible rule learning and application.
This naturally raises a crucial question for evaluating machine learning models, especially LLMs: how can we effectively measure whether model has truly learned abstract reasoning, particularly rule-inductive abstract reasoning, rather than merely memorizing patterns or exploiting superficial cues? Consider again a model solving binary arithmetic. It might achieve correct results either through a genuine understanding of abstract arithmetic rules applicable any base, or by simply memorizing common input-output patterns specific to binary operations represented with '0' and 'I', Traditional accuracy metrics alone, while useful, are insufficient to distinguish between these fundamentally different approaches. We need evaluation methodologies that can probe deeper into the nature of a model's reasoning and reveal whether it is grounded in genuine abstraction or mere memorization.
A.2. Proofs of Theorems
A.2.1. PROOF OF THEBRRM 3.7: VALIDITY OF I' FOR RULE-GIVEN POTENTIAL
Let be a sufficiently high threshold and consider model / with Abstract Reasoning Score I' 2 7
Step 1: By definition T, we have:
Σ (((an))) = 27
(10)
Step 2: For large sample size N, by the Law of Large Numbers:
p(i((c,)) g((,,,,))) E 7) 27
(11)
Step 3: Let be the abstraction mapping and Re the reasoning function. For high y:
r( 27
(12)
Therefore, high accuracy original symbolic representations implies Rule-Given proficiency.
A.2.2. PROOF OF THEOREM 3.8: VALIDITY OF A FOR RULE-INDUCTIVE ABSTRACTION
Consider model A with A < 5 and P 2
Step 1: By definition of A:
T- l' <
(13)
where MM is accuracy under symbol mapping M. Step 2: For any conerete instance E
(P(H(s))) -P( M(a))) 5
(14)
Step 3: Given r 2 7, have:
2()
M())
27-8
(15)
Step 4: Let be the abstraction mapping. For small 8:
f(c) = f(M(c) = r= Re(f(M(().r)
(16)
This invariance to symbol mapping demonstrates Rule-Induch Induutivee abstraction.
14

---

Benchmarking Abstract Reasoning in LLMs
A.2.3. PROOF OF THEOREM 3.9: SCORE RANGE INTERPRETATION
We construct and validate the mapping F.
Step 1: Define F as weighted sum:
F(T, w1 r + #2 A)
(17)
Step 2: Verify properties for extreme cases:
Case (High Ability): When r x and Rz 0:
F(T, 11 W2
(18)
Case (Medium Ability): When r 0.5 and A 0.5:
F(T, A) 0.5
(19)
Case (Low Ability): When I A
F(T, 0
(20)
Step 3: F is continuous and monotonic in I and A, ensuring smooth transitions between ability levels.
Therefore, F provides a valid mapping from scores to abstract reasoning ability.
A.3. Model Performance and Memory Dependence Table
A.4. Diagrams
4SmbmrYbni
Mupped Op
01000110 binary_and
Map
binury.and and
Ruar
Muppprr String
010001 qSmbmr/bbtt 00011111 000001:0
JQLIIQQU
Mapped Num
Num-Mapped String
JQUAQQU binary and ..JQQQQQ JAAJJQQQ
Map
01000110
Mapped String
JQNAIQQU qbnbbm/YbbD JMQOO0O JANNIQ
Figure 5. Illustration of the symbolic mapping process for operands and operators.
A.5. Symbol Mapping Protocols and Implementation
Symbol mapping is central to disentangling abstract reasoning from superficial memorization, forcing focus rule structures. We employ operand, operator, and combined remapping, as illustrated in Figure 5. To specifically assess memory dependence, Symbolic Reasoning (SR) tasks, derived from Extended Calculation (EC) tasks through these mappings, are used conjunction with EC to calculate the Memory Dependence Score (A). Performance evaluation under remappings quantifies memory dependence (A) and the robustness of abstract reasoning. The generation of remapped datasets follows a systematic procedure:
1. Candidate Symbol Pool Generation: A pool of candidate symbols for remapping is curated from the Llama-2 7B tokenizer's vocabulary. This vocabulary is filtered to retain only single-character alphanumeric tokens (c.g., 'a'-"z', 'A', '0'-99' that are treated as individual tokens). This ensures that the new symbols are basic, unlikely to carry strong pre-existing semantic meaning in complex sequences, and are distinct from structural elements like spaces or newlines. Let this set of candidate remapping tokens be denoted Swand-
15

---

Benchmarking Abstract Reasoning in LLMs
Table Model Performance and Memory Dependence Evaluation. I values represent accuracy on task categories: Basic Computation
(BC), Extended Calculation (EC), Number Base Reasoning (NBR). Math Application (MA), Symbolic Math Abstraction (SMA), Symbolic Reasoning (SR). values show performance drops under memory dependence tests.
MODEL
BC
BC
NBR
MA
SMA
SR
AVG
OP
NUM
ALL
7B-SCALK MOBELS
INTERNLM2_5-1
AL., 2024)
INTERNIM2_5.1 7--.55-7---00 C-TTTHATTTCCCCC COT
0.40
0.39
0.02
0.82
0.13
0.23
0.24
0.10
0.25
0.27
GLM-4-98-CHAT (GLM AL., 2024)
0.20
0.00
0.15
0.17
0.14
0.27
GLM-4-9B-CHAT COT
0.24
0.46
0.10
0.84
0.14
0.22
0.25
0.23
0.30
0.34
V1-1.5-9B-CHATE 16K (YOUNG ET AL., 2024)
0.53
0.38
0.03
0.29
0.11
0.14
0.19
0.09
0.36
0.38
YI-1.5-9B-CHAR
0.49
0.48
0.08
0.30
0.32
GEMMA-2-9B-IT (TEAM ET AL., 2024)
0.44
0.38
0.03
0.21
0.12
0.14
0.18
0.10
0.37
0.37
0.19
MARCO-Ol (ZHAO ET AL., 2024)
0.49
0.39
0.04
0.31
0.11
0.
0.19
0.10
0.37
0.38
MARCO-O1 COT
0.47
0.38
0.03
0.50
0.11
0.14
0.19
0.13
0.35
LLAMA-3.1-8B 1-8-111-------UCCC (DUBEY ET AL., 2024)
0.35
0.35
0.02
0.75
0.11
0.12
0.17
0.11
0.35
0.35
LLAMA-3. -88-INNTRUCCT COT
0.27
0.35
0.07
0.87
0.19
0.18
0.21
0.14
0.23
0.23
LLA.LA..AM---- (TOsHNIWAL 2024)
0.25
OPENMATH2 SAAHHH2-LLMMA, LLAMA3.1-88 cor
0.14
0.21
0.05
0.83
0.11
0.16
0.16
0.02
0.08
0.10
QWEN2.5. 70--SSRRCT (YANG Al.,, 2024)
0.37
QWEN2.5-27 1--7-INSTRUCC COT
0.57
0.55
0.31
0.91
0.13
0.27
0.34
0.23
0.34
0.40
QWEN2 MATII-B-INNSRRUS
0.49
0.34
0.09
0.95
0.12
0.12
0.18
0.15
0.27
0.27
MATH-7B-INGEE
0.19
0.11
0.02
0.94
0.01
0.05
0.07
0.04
0.06
0.08
NORMAL
0.41
0.35
0.03
0.11
0.13
0.18
0.32
COT AvG
0.36
0.37
0.10
0.82
0.10
0.18
0.21
0.15
0.25
0.27
32B-SCALE MODRLS
QWQ-32B 3-0-33---PEEIEWWW (TEAM, 2025)
QWQ-32 32B PREVIEW COT
0.78
0.83
0.53
0.95
0.13
0.40
0.50
0.31
0.50
0.58
70B-SCALE MODELS
LLAMA-3. --70--INSTRUCT COT
0.52
0.09
0.11
0.16
0.22
0.11
0.42
0.43
LLAMA-3. 700-INRUUTT
0.53
0.69
0.21
0.96
0.09
0.43
0.43
0.17
0.37
0.43
LLAMA-3. 1 -NNMOTRONN-0R TON-SRRRT COT
0.45
0.66
0.19
0.95
0.12
0.38
0.40
0.15
0.42
0.45
OPENMATH2-LLANN LLMM1I--LAM331-000
0.47
0.12
0.13
0.15
0.23
0.13
0.40
0.41
OPENMATH2 LLAMA3.1-70B
0.48
0.42
0.24
0.53
0.04
0.19
0.25
0.12
0.35
0.37
QWEN2. QWEN2.5-72B- 5-B2B- INSTRRCCT
0.64
0.49
0.29
0.68
0.11
0.18
0.28
0.07
0.42
0.43
QWEN2.5-72B-LNSTRUU
0.71
0.39
0.08
0.47
0.47
NORMAL AVG
0.53
0.15
0.12
0.16
0.24
0.41
CoT AVG
0.54
0.62
0.25
0.84
0.08
0.35
0.38
0.14
0.38
0.43
API-BASED MODELS
GPT-4O-MINI COT
0.57
0.72
0.25
0.92
0.10
0.41
0.43
0.20
0.47
0.44
GEMINI-1 1.1 5-ELLSH (TEAM AL., 2023)
0.59
0.50
0.28
0.36
0.04
0.32
0.34
0.06
0.28
0.27
DEMINI-L.5-FLLE COT
0.61
0.64
0.27
0.93
0.02
0.37
0.40
0.18
0.36
0.42
GEMINI-2 O-FLASHI-EXP
0.72
0.52
0.26
0.56
0.14
0.19
0.29
0.09
0.41
0.44
GEMINI-2. O--LLLSSE-EEPP
GEMINI- 2.0 FLASH-THINKING THINKING-ER EXP
0.80
0.60
0.76
0.95
0.12
0.20
0.39
0.10
0.48
0.51
GEMINI-248 WLA.--HHHNIINGG...CCCOOO
0.54
DEEPSCEK V3 (LIU ET AL., 2024x)
0.80
0.67
0.64
0.96
0.13
0.37
0.48
0.08
0.37
0.41
DEEPSECK V3 COT
0.79
0.66
0.50
0.96
0.12
0.29
0.41
0.15
0.40
0.47
NORMAL AVG
0.69
0.58
0.39
0.74
0.11
0.30
0.38
0.08
0.36
0.39
0.70
0.46
0.43
AGENTS FRAMEWORKS
AGENTCHATIAUTOGEN (Wu ET AL., 2023)
0.96
0.88
0.95
0.90
0.10
0.43
0.60
0.25
0.50
0.56
AL.. LIM DEBATE (DU ET AL., 2023)
0.46
0.70
0.53
0.63
0.19
0.86
0.12
0.31
0.35
0.21
0.40
0.41
NORMAL AVG
0.82
0.78
0.63
0.87
0.09
0.33
0.47
0.29
0.50
0.56
2. Identification of Original Symbols for a Task Instance: For each original task instance, consisting of few-shot examples and a question-anseee answer pair, we first identify the set of all unique non- non-whitespace characters, Unrigs present in its textual representation (both questions and answers within the examples and the target QA pair).
3. Random Bijective Symbol Mapping Establishment: random bijective (one-to-one and onto) mapping function, Maym U - Suma established. Here, Srama is a randomly selected subset Scand such that | = |Uong]- This ensures that each unique original symbol mapped unique novel candidate symbol.
16

---

Benchmarking Abstract Reasoning in LLMs
0.65
0.6
ageslchal(autogen)
0.55
gemini-2 semini-2.0-ae 2.0-flash
w-llla-ttinniinggee.. thinking exp Cot
gemini-2.0 -x...pas-eexpppppp Cot
0.5
QWQ-32B 3-ee-eeeewewwwe CoT
deepseek v3
Qwen2.5-72B Instructt CoT
react
0.45
gpt-4o-mini Coll
Llama-3.3-70BB CoT
deepseck v3 CoT
Average Score (Avg, 7)
0.4
L.ama-3. - Nemotron-700B-mme CoT
apt 4p--4o-
gemini- -2.0-200fff So--flash--iii faash-thinking--a
0.35
Il debate
Qwen2.5-7B-Instte Instrrucc CoT
gemini- Semini-1.5--lashe 1.5-
Yi- Yi-1.5-9B-Cha-11e CoT
0.3
gemini-2.0-flash-onr 0-flash-oxp
Qwen2.5-77B-Inntree
0.25
OpenMath2 LoeaMath2-LLama3,. 1-70B CoT
intemmm2 5-7b-chat CoT
Llama-3. N--emmtronn Instrutt
QWQ-32B-Prevt --ervvii
Llama-3.3 3.3maa-..-77B---- 3-770-Innttruct
Chat- Vi-. Llama-3 3.1-BB 80-88-IInstrre CoT
0.2
Marco-ol CoT
OpenMath2-Llama33.
Marco-o1 2n 1-M-5---hh7- 7B
Llama-3 1.1BBBB.BBBB---. 3.1-8B-Instruce eestrrucommma m--4 -------------- 4-9b-chatt
intermm2 5-7b-chat
Open CoT
0.15
0.1
Owen2.5-Math-7BB IB-Intruct CoT
5
10
2
0.75
0.7
0.65
0.6
0.55
0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
5
10
Memory Dependence Score (MemDep all, 6xl))
7B Fcale 32B-Scale 70B-Scale API-Based Agents
Figure Memory Dependence Average Score(Large Scale)
4. Generation of Remapped Task Instances under Different Strategies: The original task instances are then trans- formed by applying character wise substitution based on specific mapping strategies to produce remapped instances. The core strategies are:
17

---

Benchmarking Abstract Reasoning in LLMs
Table Detailed Performance Comparison: Fine-tuned Llama-3.1 Inststrrtttt Variants and Human Bascline. Accuracies percentages. (*) denotes training on unmapped symbols; (**) denotes training on fully remapped symbols. "Untrained" is the base model. "N/A" indicates data not applicable for that row/column (e.g., human baseline for fine-tuning, or models not subjected fine-tuning on these specific configurations).
DATASET
MODEL/ PARTICIPANT GROUP
UNTRAINED
FINE-TUNED (EPOCH 8)
HUMAN BASELINE
L.AMM-3.1--88 11-8--IISRRICTT DIRECT
13%
18%
LLAMA-3.1-8B INSTRUCT* --B-SSRRUU- COT
5%
7%
LLAMA-3.1 1-----.1-----SS- -N-SSRURCT DIRECT
13%
60%
LLAMA-3.1-88
COT
5%
11%
GEMINI-2, FLASILLTTIIIIIIIII
DIRECT
57%
GFMINI-2.00 FLASH FEL.AH-TTHINNE THINKING- EXP COT
46%
N/A
LLAMA-3. 1---B- -I--NSTTUCC DIRECT
27%
27% %
LLAMA-3. 3.1 --------INSSI INSTRRCTT** COT
3%
7%
LLAMA-3. -BB-INSRUUY+* DIRECT
27%
25%
F.X.D..BK..DRAA..TT..ATTEE
LLAMA-3. BB--IIITRUCTT CoT
3%
2%
47%
GEMINI-2.0 FLA.H-LHHHHKIIEE
DIRECT
19%
N/A
GEMINI-2. 0- TEASH-THHNKING-- COT
17%
N/A
AD0.2ASE3.RAY.DAAAASS
GEMINI-2.0 C-FLLSH--TIII THINKN--INNNNG-EEXX
75%
N/A
87%
The original task instance is used as-is. This serves the baseline for calculating T. Combined Remapping (All Symbols): The mapping Msy (derived from all unique characters Unrig) is applied to all characters in the task instance's 's examples and QA pair. This generates the fully remapped dataset variant used for one type of A calculation.
Raw (No Remapping):
Operand-Specifie Remapping (Number/Value Symbols): First, unique characters constituting only the operands (e.g.. '0', '1' in bit strings; digits in numerical operations) are identified from Unrig. let this set be Uoperand- A distinct random bijective mapping Moperand Uoperand + 5 is created. This mapping is then applied only to the operand characters within the task instance. Operator symbols remain in their original form.
Operater-Specific Remapping (Operation Symbols): Similarly, unique characters representing operators (e.g., "binary and', '4', or their symbolic representations in the examples) are identified, let this set be U Upperntor- A distinct random bijective mapping Moperatore Uoperator - S is created. This mapping is applied only to the operator characters. Operand symbols remain unchanged.
For all remapping strategies, the underlying problem structure and the rules implied by the examples are preserved; only the surface symbolic representation is altered.
5. Dataset Assembly:
By applying these strategies, we generate multiple versions of each sub-dataset: a raw version, a fully remapped version, an operand- version, and an operator-remapped version. These variants allow for the calculation of T (from raw performance) and different facets of Á by comparing performance on raw versus remapped datasets.
This systematic remapping allows us to quantify how much a model's performance relies on specific familiar tokens versus its ability to generalize to abstract patterns represented by novel symbols.
A.6. Dataset Details
Our benchmark comprises 82 sub-datasets, categorized across six main task categories: Basic Computation (BC), Extended Calculation (EC), Number Base Reasoning (NBR), Math Application (MA), Symbolic Math Abstraction (SMA), and Symbolic Reasoning (SR). With the exception of the Math Application (MA) category which utilizes the GSM8K dataset, each sub-dataset contains 96 examples, resulting in a total of 9095 samples across the entire benchmark. During dataset construction, ensured diversity in task instructions and samples, and meticulously excluded potential duplicate phrasings to prevent models from solving tasks via superficial pattern matching. The naming convention for sub-datasets indicates key variations: prefixes like "var len' or 'fixed len' denote variable or fixed operand lengths, respectively; 'chat" signifies a conversational prompt format. The core of the filename often indicates the specific operation type (e.g., 'bit' for bitwise operations, 'str' for string manipulations, 'strop' for specific string operations). Furthermore, suffixes such as 'raw' identify tasks using original, unmapped symbols, while 'op map', 'num map', and 'all map' specify whether symbol remapping
18

---

Benchmarking Abstract Reasoning in LLMs
applied to operators, operands (numbers), both, respectively. A detailed list of sub-datasets within each task category is provided below:
A. BASIC COMPUTATION (BC) DATASETS
chat_add_datasee
chat_div_dataser
chat_sub_datasee
chat_mul_dataset
A.6.2. EXTENDED CALCULATION (EC) DATASETS
par_le__cccc_liiss_tt______ddddd
five__ien_chat_btt__eee.dddddd
fixe___lenccaat_ttt_eeeedddadd
va._een.chaa_bbttoerrre.d..dd
va.__ea_ccha__ttt_aaaaaattttttt
aag_len.cchtt_llst_reeeeeettttt
chat_squurre_datasee
var_lenjcaa_jat__aaacattattt
va._lenncche__ddcee_reeeaaaaedd
var_jee__cha__ee___aa_ctttttttt
fixe__Jen_ccat__stco___eee.aaeed
A.6.3. MATH APPLICATION (MA) DATASETS
dataset_gsmak (1319 samples)
A.6.4. NUMBER BASE REASONING (NBR) DATASETS
Laa__add_base3_raw
chet_odd_aase__raa_dtteeer
chat_baaee_raw_dattastt
chat_suhh_baaee__aa__taaat
chat_mal_jbsee__aaadaaaeee
chat_bbasee_rew_ddattsttt
chat__maljaase@_aaavaaaaaeee
chat_poaee__raw_Gattaett
chat_ace_jaae5__aaadaaeeee
chet_acl__aaas3__ra__ataaee
chat_mal_baase_saaadatteeer
Chtt_sub_jaaee5_aaaaaaaaaas
A.6.5. SYMBOLIC MATII ABSTRACTION (SMA) DATASETS
hnt__qaadratiic_dttasse
chat_trrlangle_wve__ddaee
dataset
chat_sewtcoth__wwe_ddaaaaa
blat_sqquare_wave_dttaer
chat_cosiin__dataset
chat_linear_dataset
19

---

Benchmarking Abstract Reasoning in LLMs
chat_sine_dataser
A.6.6. SYMBOLIC REASONING (SR) DATASETS
fike__Jen__hattsstoo____ettttt
fiee__Len_chat_bbttoo__.ctttee
va__een_caa__istt_onn_______ttt_ at
var_Jee_ccht__bit_ddttteett
Bas_le___cha__ai__hhhff________
dataset
var_len_cha__itt__o_ddaaattt
var_een_chat_lisssoop_aaaaeee
va__eeh_chat_jiss__nn_aaaaaaaa
vor_len_chat_eet__atteeet
aa__een_chat_Jiss__nnt_taaeeee
aa._een_chha_llltcp_____aaaae
v
Fixe__eenn_caat__s__nnnaaaccee_
var_len_chat__srrco.ddaaaa
var_leencche__ttt_oo_.daaatt
var_len_chat_llss__aaaattt
fixe__ien_ccaa___ttr_datases
vaj_len_cche___ee_nnnnnaaaatttt
fiee__een_chaa_jiioq_aaaattt
pat_le__chaa__stccp_oo__aaatdd
va__lenncche__bttt_nnnn_ddeeeee
fixee_ieen_chatttouuttttttteeee
tixe__llenccaat_ccccoood.dtttt
rixe__ean__catt_ttttooooo..aaae
vat_leei_chh__ssttqpnnn_ddadeee
Fixee_Jin__chtt_btt__aaaaassss
var haa__ee__hha__es__opdddttsss
fivee__een_chaa_pbttoooppppaade
var par_len__cae Seee__hhat_ssee var_Len_chat_stt_dddaaee dataset
fi fixee__een_che___uuattto___._...
Total Samples: 9095
Dataset Structure: Each dataset consists of input-output pairs designed to evaluate specific abstract reasoning skills as detailed Section 4. The input and output formats are consistent across all sub-datasets within each task category, ensuring a standardized evaluation framework. The 'Avg' column in Table represents the average of the Abstract Reasoning Score (T) across all sub-datasets within the corresponding task category.
20

---

Benchmarking Abstract Reasoning in LLMs
A.7. Example Demonstrations
Here we provide example prompts and instances from our benchmark illustrate the task format and the symbol mapping methodology.
A.7.1. EXAMPLE FROM EXTENDED CALCULATION (EC) CATEGORY
The task is to identify patterns and discover rules from the provided examples, then answer a question. The symbols in the question may not have their usual meanings, so carefully analyze the rules and expressions before providing your final answer in the format: "Answer: The answer is {your answer)."
Prompt:
Examples:
Question: 01000110 binary and 0001 1111
Answer: The answer is 000001 10.
Question: 0001 100 binary_am and 00010001
Answer: The answer is 00010000.
Question: 01011110 binary_and and 00001 101
Answer: The answer 00001 100.
Question: 00100111 binary_and 011001 Answer: The answer is 00100111.
A.7.2. EXAMPLE FROM SYMBOLIC REASONING (SR) CATEGORY WITH SYMBOL MAPPING
Prompt: The task is to identify patterns and discover rules from the provided examples, then answer question. The symbols in the question may not have their usual meanings, so carefully analyze the rules and expressions before providing your final answer in the format: "Answer: The answer is {your answer)."
Examples:
Question: JQUJQQQ q5nbmyYbnD) YbnD JIIQQQQQ
Answer:
The answer is JJJJJQQJ.
Question: JIJQQQU 45nbmv YbnD JIJQIJQQ
Answer: The answer is JIIQJJJJ
Question: JQIQQQQI S5nbmvYbnnl JIJJQQIQ
Answer: The answer IIJJQQUJ
Question: JJQJIQQQ q5nbmv YbnD JQQIIQQQ z Answer: The answer is JJQJIQQQ
A.8. Example Instances for Task Categories
Here we provide example instances for each task category our benchmark, drawing from the benchmark document provided:
BB (Basic Computation):
Addition: '27 15817
21

---

Benchmarking Abstract Reasoning in LLMs
Subtraction: '100-25=??
Multiplication: '12 *
Division: '24/8= ==/8=?
EC (Extended Calculation): This category includes a variety of extended computational tasks:
Square Root Calculation: 'sqrt(625)
Bitwise Operations:
Binary AND: '010001 10 binary_ and 00011111=??
Binary OR: '010001 binary or 00011111
Binary NOT: "binary_nor 01010101
Bit Shift Operations:
Logical Left Shift: '00000110 bit_shift_leff 2
Logical Right Shift: 100000110 bit.shift.right 2 =
Circular Right Shift: '00000110 10 circular.right.she 1 ?"
Bit Manipulation Operations:
Check Bit: '010001 10 check_bi == (Check if bit at position is set)
Set Bit: 101100010 set bit 0 = ?" (Set bit at position to 1)
Toggle Bit: '00010100 toggle_bit 6 = ?" (Flip bit at position 6)
String Manipulation:
Reverse String: 'reverse(" 'reverse('algorithmi) 'algorithm'')
? =
Concatenate Strings: 'concatenate( 'hello', 'world') =
Repeat String: 'repeat('go "go'' ?
Get String Length: get length( (benchmarkingg)=
Substring Containment (in order): sbblehdbbgghhlegaaacccchhhhahbbb contains(in order) bffd ?
Set Operations:
Difference: "difference(O, 2, 4, 0, 1, 4) =
Union: 'union(a, b, c, c, e)
Intersection: 'intersection(1, 2. 3, 3, 4, 5) =
List Operations:
Sort List: 'sort[[5, 2, 8, 1, ="
Filter List: 'filter[[I, 5, 10, 3, 8). 5) = (Filter elements greater than 5)
Deduplicate List: 'deduplicate((a, b, c, b, d])
Maximum Value in List: 5max 23, =5))=? ?
Minimum Value in List: 'min([12, 23, 15))
Median Value in List: 'median[[3, 1, 1,5, 2, 6]) ?
Mode Value in List: 'mode([a, b, c, b, a, a, d)) =
Date Calculation:
Days Between Dates: between daaieemnddats(222,,,,.. 07, 29), [2021, 10, 31)) ?
NBR (Number Base Reasoning):
Ternary Addition: '2200102 (base3) + 11100111 (base3) = (base3)'
Quaternary Subtraction: "321 (base4) 13 (base4) ? (base4)'
Quinary Multiplication: '23 (base5) (base5) = ? (base5)'
Base Conversion: '25 (base10) to base 3 = (base3)'
MA (Math Application):
GSM8K style problems, for example: "If Maria buys apples and each apple costs $0.50, and she also buys banana for $1.00, how much does she spend in total?"
SMA (Symbolic Math Abstraction):
22

---

Benchmarking Abstract Reasoning in LLMs
Infer function type and parameters from input-output pairs:
Linear Function: Input-Output pairs: (I, 5), (2, 8), (3, 11). Question: fun(4) = (Function is f(x) = 3x + 2)
Quadratic Function: Input-Output pairs: (1, 2), (2, 7), (3, 14). Question: fun(4) = (Function is f(x) = x2
Exponential Function: Input- pairs: (1, 6), (2, 18), (3, 54). Question: fun( 4) (Function is f(x)= =
3&)
Logarithmic Function: Input-Output pairs: (1, 0), (e, a), (e2, 2a). Question: fun(e3) (Function is f(x)
In(x))
Sine Function: Input-Output pairs: (0, 0), (z/2, a), (m, 0). Question: fun(3r/2) (Function is f(x) sin(x))
SR (Symbolic Reasoning):
For the Symbolic Reasoning (SR) dataset, utilized tasks from the Extended Calculation (EC) category, excluding tasks involving square root calculations (sqrt) and date-related operations (data).
A.9. Benchmark Task Generation Details
Definition A.1 (Task Generation Function). For each task category c, the generation function Ge defined as:
Ge R x P 9:)1
where:
R the rule space.
P is the parameter space for task configuration.
(zi, v:) are input-output pairs satisfying y, =R(((z))r) for E R.
A.10. Illustrative Case Studies: Success and Failure in Abstract Reasoning
To further illustrate the performance characteristies of LLMs our benchmark, this section presents specific studies: one failure and one success. For brevity, the few-shot examples provided in the full prompt omitted here; focus on the final question posed, the ground truth, the model's full 'raw output' (which includes its Chain-of-Thought process), and an analysis of its reasoning.
A. 1. FAILURE CASE: NON-DECIMAL ARITHMETIC (BASE-5 ADDITION)
Task Context: The model was tasked with performing addition in base-5, with explicit hints and examples indicating the base of operation. This tests the model's ability generalize arithmetic principles to non-decimal systems.
Model: DeepSeek-Ve V3 (with CoT prompting)
Dataset: chat chat..dd.basse5..a..ttte..
Question Posed (final part of prompt): \nHint: This is base 5 operation ierration. ion, .nn.Anuusstion: 411
421200 i)nnnn (Examples omitted)
Ground Truth Answer: 422111 (base 5)
Model's Extracted Final Answer: 422211 (base 5)
Model's Full fraw output':
To solve ( 411 421200 A) in base 5, we follow these steps:
## Step 1: Align the numbers by their place values
align the numbers so that the digits are the correct place value columns:
23

---

Benchmarking Abstract Reasoning in LLMs
421200 411
### Step 2: Add the digits column by column,
starting from the rightmost column (units place)
We add the digits in each column, keeping mind that this is base 5, any sum greater than equal to 5 will carry over to the next column.
1. Rightmost column (units place)
0 + 1 V).
Write down *****.
2. **Next column (fives place) *** \ 0 +
\).
Write down **1**.
3. **Next column (twenty- fives place) :**
( 2 + 4 V)
Since 6 is greater than or equal to 5, we subtract and carry over to the next column. Write down **lxx and carry over **1**.
4. **Next column (one hundred twenty-fives place) :**
\ 1 0 V), plus the carry-over of 1 gives + = V.
Write down **2x*.
5. **Next column (six hundred twenty-fives place) ***
\ 2 0 \).
Write down **2**.
6. **Next column (three thousand one hundred twenty-fives place) :** \( 4 + 0 \).
Write down **4**.
7. **Leftmost column (fifteen thousand six hundred twenty-fives place) \< 4 + 0 V.
Write down **4**.
### Step 3: Write the final result
Combining all the digits from left to right, the result is:
422211***
### Final Answer:
The answer is **422211**.
Analysis of Model's Reasoning: The model's 'raw output' indicates attempt follow base-5 addition procedure. It conrectly identifies the need for carrying when column sum is 2 5. For instance, in "Step 2, part 3," it calculates 2 + 4 = 6. which becomes 'I' with carry of 'I' in base 5. The critical error occurs in the handling of place values for numbers of differing lengths. While '421200' (base 5) a six-digit number and '411' (base 5) is three-digit number, the model's "Step 2, part 7" incorrectly processes the most significant digit '421200' (the leftmost '4') as if it were a seventh digit to be added to an implicit leading zero from the shorter number. This leads an erroneous extra '4' at the beginning of its
24

---

Benchmarking Abstract Reasoning in LLMs
sum, resulting in '422211" instead of the correct '422111'. This case demonstrates failure in accurately abstracting and applying place value concepts in non-decimal system, when the fundamental carry logic is partially understood. The CoT, while outlining steps, did not prevent this structural misunderstanding
A. A.10.2. SUCCESS CASE: SYMBOLIC REASONING WITH REMAPPING (BITWISE AND)
Task Context: The model was presented with few-shot examples where two 8-character strings composed of novel symbols ('Z' and '3') were combined with a distractor operator string to produce an 8-character output string. The task required inferring both the meaning of the novel symbols and the underlying logical operation.
Model: Soomini-..--Pash---hhiiingg Exp (with CoT prompting)
Dataset: Eixed_.en.chat..itt.ccr
fully remapped bitwise operation task)
Question Posed (final part of prompt); Question: 27333273 LHWFRwgFWr 77z37237 GlninLet's think step by step. |n|n (Examples omitted)
Ground Truth Answer: 272377Z2
Model's Extracted Final Answer: ZZz3ZzZZ
Model's Full 'raw output':
Answer: The answer is 22Z3ZZZ2.
*xStep-by-step reasoning:**
1. **Convert 'z' and '3' to binary digits:** Treat '2' and '3' as 1.
-First string: 22333223 becomes 00111001.
-Third string: AZZ37732 becomes 00010010.
2. **Perform bitwise AND operation:**
Apply the bitwise AND operation to each corresponding bit of the binary representations the first and third strings.
00111001 (First string)
AND
00010010 (Third string)
00010000 (Result)
Position 1: AND
Position 2: 0 AND 0
Position 3: 1 AND
0
Position 4: 1 AND
1 =
Position 5: 1 AND
0 =
Position 6: 0 AND
0
Position 7: AND
Position AND
3. **Convert the binary result back to '2' and 13' Convert the binary result 00010000 back to string of z and '3's, where 0 is '2' and 1 is '3. 00010000 becomes 22232Z22.
4. +*Compare with the provided answer:**
25

---

Benchmarking Abstract Reasoning in LLMs
The derived answer 22z3zzz2 matches the provided answer for the question 22333Z73 LHWFRwgFWr ZZZ3ZZ3Z G.
Final Answer: The final answer is $ \boxed [22232222
Analysis of Model's Reasoning: The model's 'raw output' clearly articulates a successful multi-step abstract reasoning process:
1. Symbolic Abstraction: In "Step 1," the model correctly infers the semantic mapping of the novel symbols: 'Z' to binary "O' and '3' to binary 'I'. It then accurately converts the two relevant input strings ("ZZ333ZZ3' and 'ZZZZZZ3Z') into their binary representations ('00111001 and '00010010', respectively). The distractor elements 'LHWFRwgFWr" and 'G' are correctly ignored.
2. Operational Inference and Execution: In "Step 2," the model correctly identifies the underlying operation as a bitwise AND applied to the two abstracted binary strings. then performs this operation flawlessly, yielding the binary result '00010000'.
3. Reverse Symbolic Mapping: In "Step 3," the model converts the binary result back to the original symbolic domain, correctly translating '00010000" to '77737777'
This case exemplifies successful abstract reasoning. The model was not merely pattern matching surface tokens but demonstrated an understanding of the underlying logical structure by (a) mapping novel symbols to a known representational system (binary), (b) inferring the correct logical operation from examples, and (c) applying this inferred rule to new inputs. The CoT output provides transparent evidence of this robust reasoning process.
A.11. Supplementary Experimental Details
This section provides further details on the Large Language Model (LLM) evaluation setup, the fine-tuning experiments, and the human baseline evaluation.
A.11.1, LLM EVALUATION SETUP
The main LLM evaluations conducted utilizing two distinct server configurations, selected based model scale and computational needs: server equipped with 8 NVIDIA GeForce RTX 3090 GPUs (24GB VRAM each), and another with 8 NVIDIA A800 GPUs (80GB VRAM each). For inference, we generally employed a generation configuration with temperature of le-7 and a maximum of 2096 new tokens. The inference batch size was adjusted according to available GPU memory and model size, prioritizing model-specific default generation parameters where applicable.
A.1. FINE-TUNING SETUP FOR LLAMA-3. 3...8..---B.... -SB--ISSTUUCT
To investigate the impact of training data on abstract reasoning with remapped symbols, we fine-tuned the Llama-3, 3.1-a--..1-B-... Instruct model. Input prompts and their corresponding answers were formatted continuous sequence, specifically structured as <>> INST] {prompt} 
 <</>> We utilized the Tokennke from the "meta- Ilama/Llama-3,, 1--8BB Instrucct" pretrained model, configuring it with padding on the left side and setting the pad token to be the tokenizer's is end-of-sequence (EOS) token. All inputs were tokenized to maximum length of 2048, with truncation and padding applied as necessary.
Two distinct training data configurations were employed. The "Unmapped Symbols Training (*)" involved fine- tuning the model on 2,000 samples generated using original, unmapped symbols, structured similarly to our "fixed_len Leecchtten_cht_____ schat_bit.ttee raw dataset". Conversely, the "Fully Mapped Symbols Training (**y" fine-tuned the model on 2,000 samples where symbols (both operands and operators) were systematically remapped, mirroring the structure of our "fixed len chat bit dataset".
Key hyperparameters, based on our experimental script, included a per-device batch size of 4, 2 gradient accumulation steps, the Paged AdamW 8-bit optimizer, learning rate of 2e-5, and cosine learning rate scheduler with a warmup ratio of 0.03. BF16 precision was used, the seed was set to 42, and gradient checkpointing was enabled.
26

---

Benchmarking Abstract Reasoning in LLMs
Post fine-tuning, the performance of these models was assessed remapping structure encountered during the 'Fully Mapped' training phase) and the "fixed len chat sur dataset" (which presented an unseen remapping structure and task type), to evaluate both learning and generalization. The Chain-of-Thought (CoT) variants mentioned in our earlier rebuttal refer to the application of CoT prompting strategies during the inference phase with these already fine-tuned models; the fine-tuning data itself did not explicitly include CoT examples. Detailed results, encompassing pre-fine-tuning (untrained) baselines and post-fine-tuning performance for both direct prompting and CoT variants, are collated Table
datasets: the
_it_aset (where the
A.11.3. HUMAN BASELINE EVALUATION PROTOCOL
To establish human performance baseline for comparative analysis with LLMs, recruited four undergraduate students, each with background in computer science. These participants were tasked with subset of 16 samples from each of the following datasets: the "fixed www.eenn.ooo .chat_bitttt dataset" (for remapped bitwise operations), the "fixed Ien son_chatsstt dataset" (for remapped string operations), and the "add base3 raw dataset" (for non-decimal arithmetic, specifically base-3 addition with original symbols).
During the evaluation, participants received the identical problem descriptions and examples that were provided to the LLMs in their respective evaluation settings. They were instructed infer the underlying rules from these provided examples and subsequently solve the posed questions. It is important to note that explicit training instruction regarding the specific symbol mappings was given to the participants beyond what could be deduced from the in-prompl examples. Performance was quantified by calculating accuracy based on the correctness of their final answers. The aggregated human performance across these tasks is detailed and compared with LLM performance in Table 4.
27


Reviews:
[{"review_id": "DmsA7ffAkn", "reviewer": "Reviewer_gYuP", "summary": "The goal of the paper is to evaluate the abstract reasoning capabilities of LLMs. The paper points out flaws with two existing benchmarking paradigms: The symbolic reasoning benchmarks like GSM8K risk memorization since the models could be (inadvertently) trained on these benchmarks. The visual abstract benchmarks like ARC are not well suited for LLMs due to their visual nature. The paper first lays out what it means by reasoning: abstraction, where the model identifies patterns from various inputs, followed by reasoning, where the model applies consistent rules to these abstractions to arrive at some outputs. The paper builds a symbolic benchmark and shows gaps in reasoning abilities of LLMs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1. Test Set Sampling and Duplicates\n\n1. **Test Set Distribution:**      \n   Uniform distribution.\n\n2. **Real-World Distribution and Correlation:**  \n   No, it is not sampled from a real-world task-based distribution such as SAT problems. However, our benchmark shows strong correlation with other benchmarks. For example, if we only consider the models tested in our paper (excluding agents):  \n   gemini-2.0-flash-thinking-exp-01-21* achieved rank 1 with the highest Γ score in our benchmark, and on LiveBench its average reasoning score reached 78.17, outperforming all other models.  \n   qwq-32B-preview* achieved scores comparable to 72B and API models in our benchmark, with a LiveBench average reasoning score of 57.71.  \n   Qwen2.5-72B-Instruct consistently outperforms competing models both in our benchmark and on the Korean SAT LLM Leaderboard baseline.\n\n3. **Elimination of Duplicates:** \n   No, duplicate phrasings have been strictly eliminated.\n\n---\n\n### 2. Symbol Remapping and Its Rationale\n\n1. **Performance Consistency After Remapping:**  \n   Yes. A truly abstract model should maintain comparable performance post-remapping. For instance, GEMINI-2.0-FLASH-THINKING-EXP-01-21 drops by 0.40 on the `fixed_len_chat_bit_dataset` after remapping, indicating sensitivity to symbolic changes. In contrast, four computer science undergraduates scored nearly identically (1.00 and 0.97).\n\n2. **Humans and Non-Decimal Arithmetic:**  \n   \n   No. Although non-decimal bases may pose challenges, humans can perform arithmetic in various numeral systems once abstract rules are applied. In our test of 16 samples from the `add_base3_raw_dataset`, undergraduates achieved 0.87 accuracy, while *GEMINI-2.0-FLASH-THINKING-EXP* reached 0.75.\n\n---\n\n### 3. Validity of the Metric F(Γ, ∆)\n\n1. **Properties of a Valid Metric:**  \n   A valid metric should be monotonic (increasing with higher Γ and decreasing with higher ∆, sensitive (proportional to changes), continuous, and normalized within [0, 1].\n\n2. **Our Metric:**  \n  Defined as  \n  F(Γ, ∆) = w1Γ + w2(1 − ∆)\n  it satisfies these properties.\n\n3. **Justification:**  \n   Our metric inherits the desirable mathematical properties documented in the literature (e.g., Sokolova and Lapalme).\n\n---\n\n### 4. Multimodal Considerations\n\n1. ARC work for multi-modal LLMs  \nYes. While ARC is designed for 2D visual inputs and can potentially benefit multimodal LLMs. For example, Align-DS-V—an experimental vision-language model derived from DeepSeek-R1-Distill-Llama-8B—achieved a score of **40.5** on the ARC-Challenge (5-shot), compared to **21.4** from DeepSeek-R1-Distill-Llama-8B.\n\n2. Consider multimodal datasets\nOur work complements ARC by isolating the text modality to evaluate abstract reasoning in LLMs. While multimodal LLMs are emerging and humans naturally integrate visual and linguistic cues, most LLMs excel at language processing. Focusing exclusively on text allows us to establish a clear baseline for abstract reasoning without the added complexity of visual data.\n\n---\n\n### 5. Applicability of ConceptARC to LLMs\n \nLLMs are optimized for one-dimensional text, not two-dimensional spatial data. Our experiments show that even when Arabic numerals and simple graphics are array-formatted, LLMs struggle with 2D spatial properties, making ConceptARC less suitable for assessing textual reasoning.\n\n---\n\n### 6. Minimality in Concept Representations\n\nOur abstraction mapping process as an information compressor that inherently normalizes equivalent expressions without requiring an explicit minimality algorithm.\n\n---\n\n### 7. Addressing Uncertainty in Reasoning\n \nThe phrase “likely to bark” in our cognitive example was intended solely as an illustrative simplification. Our framework currently outputs deterministic results post-abstraction; future work will integrate probabilistic models to handle inherent uncertainties.\n\n---\n\n### 8. Clarification on Eq. 7's Probability\n\nIn Theorem 3.7, the probability is computed over the test set T, where each task is independently sampled from a uniform distribution over the task space.\n\n---\n\n### 9. Tokenization and Long Symbolic Strings\n\nSequences like “JJJQQQQQ” may split into multiple tokens by the tokenizer; however, modern LLMs handle these effectively. In our benchmark, the majority of token counts for such symbolic strings are below 5k—well within the 8K-token (or larger) context window of current models. \n\n---\n\n### 10. Clarification of Dataset Names\n\n- var_len: Variable operand lengths.  \n- chat: Conversational style prompts.  \n- bitop: Bit manipulation operations.  \n- raw: Original symbols without remapping.\n\nDetailed descriptions and examples will be added to Appendix A.6.\n\n---\n\nWe appreciate the reviewer’s feedback and hope that these clarifications, along with our proposed modifications, demonstrate the robustness of our theoretical framework and benchmarking strategy.\nThank you very much for your consideration.\n\nSincerely,  \nThe Author Team", "claims_and_evidence": "Yes. The main claim of the paper is the design of a formal setup for measuring abstract reasoning abilities of LLMs. The paper does a fairly good job of building this setup. The paper also does quite well in grounding this setup in existing literature in cognitive sciences.", "methods_and_evaluation_criteria": "Yes. The benchmarks are math related but do provide coverage across a large number of mathematical operations (Appendix 6).", "theoretical_claims": "I went over the proofs in Appendix 2. While the proofs algebraically seem correct, I think they need some work in connecting the algebra to the application. The paper should specify what exactly the probability distributions in these proofs are, how we sample from them, and what the properties like validity in Eq. 9 are.", "experimental_designs_or_analyses": "I looked at the dataset design in Appendix A6. The datasets are relevant. However, the paper should add details and examples from these datasets. For instance, just by reading the name `var_len_chat_bitop_raw_dataset`, it is difficult for a reader to infer exactly what this dataset is and what the precise prompts used here were.", "relation_to_broader_scientific_literature": "The paper does a good job of framing itself w.r.t. the existing literature on LLM benchmarking.", "essential_references_not_discussed": "Not that I could tell.", "other_strengths_and_weaknesses": "## Strengths\n\n1. The fact that the paper formally defines the basic building blocks, e.g., concepts, abstraction and reasoning is quite useful and could lay a strong foundation for follow ups in this area.\n2. The paper is quite well-written. Formalism (e.g., Definitions in Section 3.1) is followed up with concrete examples.\n\n\n## Weaknesses\n1. Definition 3.5: The paper should explicitly mention the probability distribution the test set T is samples from. Is it sampled from some task based distribution in the real world (e.g., SAT problems)? Could it contains duplicates that are the same problem but phrased differently?\n2. Line 233: The paper should spend some more time on the validity of these alterations. Should we expect the same performance after altering 0/1 to A/B? One could argue both ways. For instance, humans can perform arithmetic in base 10 much more easily than in base, say 89. Does this different make humans poor abstract reasoners?\n3. Theorem 3 proof in Appendix 2 needs some work. The proof verifies the properties for some extreme cases and shows some monotonicity. How does adherence to these properties make the metric in Eq. 9 a “valid” metric. What properties do we want in a valid metric anyway?", "other_comments_or_suggestions": "None", "questions_for_authors": "1. ARC doesn’t work for LLMs, but would it work with multi-modal LLMs? Why not test multi-modal LLMs given that they are becoming quite ubiquitous. Also, humans learn from both language and vision. So why not consider multimodal datasets?\n2. Line 113: “However, these often lack direct application to LLM abstract reasoning evaluation”. How does ConceptARC lack application to LLM abstract reasoning?\n3. The same concept could be described in many different words. For instance, “four-legged”, “having four legs”, etc. Does the paper enforce some kind of minimality constraints on the concepts, e.g., all equivalent expressions of a concept should be mapped to a single word pr phrase.\n4. Line 146, second column: On “For instance, with the abstract concept, …”. The example mentions that “dog is likely to bark”. The word “likely” here implies some form of uncertainty in the behavior, e.g., the dog might bark in 80% of cases upon seeing another dog. How does the framework plan to quantify this uncertainty. Should this quantification be a part of the rules?\n5. Theorem 3.7: What is the probability in Eq. 7 computed over?\n6. Line 913: The strings here like “JJJQQQQQ” seem quite challenging for a LLM tokenizer, and will lead to a large number of tokens as compared to natural language. Given the tokenization issues, should we expect LLMs to be able to solve the resulting problems?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jzvc3oOUaf", "reviewer": "Reviewer_GRws", "summary": "This paper presents a theoretically grounded benchmark to evaluate abstract reasoning in Large Language Models (LLMs). It defines abstract reasoning as extracting essential patterns and applying consistent rules to these patterns. Two metrics, Γ (Abstract Reasoning Score) and ∆ (Memory Dependence Score), are introduced to measure reasoning ability and distinguish abstraction from memorization. Evaluations across various LLMs reveal limitations in non-decimal arithmetic and symbolic reasoning, as well as significant memory dependence. The findings highlight the need for improved abstract reasoning capabilities in LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Relevant References\n\nThank you for highlighting these references. Chollet (2019) is already cited (see Introduction). In the revised version, we have added:\n\n- **Garcez (2020), *Neurosymbolic AI: The 3rd Wave***  \n  This work highlights the limits of connectionist methods and supports our discussion on transitioning from memorization to true abstraction.\n\n- **Webb et al. (2023), *Emergent Analogical Reasoning in Large Language Models***  \n  This study provides empirical evidence on zero-shot analogical reasoning and contextualizes our findings on performance drops under symbol remapping.\n\n---\n\n## 2. Analysis of Training Data Configurations\n\n### Experimental Design Overview\n\nTo further evaluate the influence of different training data configurations on abstract reasoning, we tested two models—**Llama-3.1-8B-Instruct (fine-tuned)** and **GEMINI-2.0-FLASH-THINKING-EXP-01-21**—and also tested human participants.\n\n**Training Data Configuration:**\n\n- **Unmapped (*) Configuration:**\n  The training data is generated using a random seed to follow the same distribution as the `fixed_len_chat_bit_raw_dataset`. No symbol mapping is applied.\n- **Fully Mapped (†) Configuration:**\n  The training data is generated using a random seed to follow the same distribution as the `fixed_len_chat_bit_dataset`. Symbol mapping is applied.\n\n**Test Benchmarks:**\n\n- We test on two mapped datasets:\n  - `fixed_len_chat_bit_dataset`\n  - `fixed_len_chat_str_dataset`\n\n**Training Setup:** \nTraining was performed using a single NVIDIA A800 80GB GPU, with 2,000 training samples over 8 epochs, a batch size of 8, cosine learning rate scheduling (with 3% warm-up), and using bfloat16 mixed precision.\n\n### Experimental Results and Analysis\n\nBelow are our experimental results:\n\n| Model                                   | Untrained | Epoch 8 |\n|-|-|-|\n|  fixed_len_chat_bit_dataset|||\n| Llama-3.1-8B-Instruct*|0.13|0.18|\n| Llama-3.1-8B-Instruct cot*|0.05 |0.07|\n| Llama-3.1-8B-Instruct†|0.13|0.60|\n| Llama-3.1-8B-Instruct cot†|0.05|0.11|\n| gemini-2.0-flash-thinking-exp-01-21|0.57||\n| gemini-2.0-flash-thinking-exp-01-21 cot|0.46||\n| Human|0.97||\n| fixed_len_chat_str_dataset|||\n| Llama-3.1-8B-Instruct|0.27|0.27|\n| Llama-3.1-8B-Instruct cot|0.03|0.07|\n| Llama-3.1-8B-Instruct| 0.27|0.25|\n| Llama-3.1-8B-Instruct cot| 0.03|0.02|\n| gemini-2.0-flash-thinking-exp-01-21| 0.19||\n|gemini-2.0-flash-thinking-exp-01-21 cot|0.17||\n| Human|0.47||\n\n\n\n**Key Observations and Conclusions:**\n\n- Under the **unmapped configuration**, the performance improvement is marginal.\n- For the **fully mapped configuration**, the Llama-3.1-8B-Instruct model shows significant improvement on the bit dataset (from 0.13 untrained to 0.60 by Epoch 8). In contrast, the **Llama-3.1-8B-Instruct cot†** shows improvements from **0.05** to **0.11**. Analysis of the Chain-of-Thought outputs indicates that the model still largely imitates the training examples rather than explicitly inferring and applying abstract rules.However, the improvement on the string dataset is limited, suggesting that the rules learned under remapping do not generalize well across different symbolic representations.\n- **Human performance** exceeds that of the models, further underscoring that current LLMs heavily rely on memorized associations rather than fully extracting and applying abstract rules.\n\n---\n\n## 3. Comparison with Human-Level Abstract Reasoning\n\nWe conducted a supplementary experiment with four undergraduate computer science students who evaluated the same tasks:\n\n- **Bit Dataset:** Human participants achieved an accuracy of 0.97.\n- **String Dataset:** The overall human accuracy was 0.47.\n\nThis direct comparison clearly demonstrates that even with extensive training, LLMs continue to underperform relative to human abstract reasoning capabilities, reinforcing the need for further improvements in model design and training.\n\n---\n\n## 4. Detailed Benchmark Implementation\n\nWe agree with this recommendation and will revise Appendix A.7 in a future version to include detailed examples demonstrating the symbol remapping process and specific task configurations. Furthermore, we commit to releasing the complete source code, which will comprise:\n\n- A dynamic symbol mapping tool with customizable remapping rules.\n- Code templates for generating custom datasets across all six defined task categories.\n- An automated evaluation pipeline that computes both Abstract Reasoning Score (Γ) and Memory Dependence Score (∆), ensuring consistent answer matching.\n\n---\n\nOnce again, we thank the reviewers for their highly constructive feedback and for helping us identify areas for clarification and improvement. We believe that the revisions and additional experiments described above have significantly strengthened our work, and we appreciate your careful consideration of our manuscript.\n\nSincerely,  \nThe Author Team", "claims_and_evidence": "The claims are well-supported by the evidence presented. The authors provide a rigorous theoretical framework, introduce novel metrics, and conduct evaluations to demonstrate the limitations of current LLMs in abstract reasoning. The findings somewhat highlight key areas for future research.", "methods_and_evaluation_criteria": "The methods and evaluation criteria in the paper are well-suited for assessing abstract reasoning in LLMs. The theoretical framework and metrics measure abstraction versus memorization. The benchmark, with systematic symbol remapping and diverse tasks, rigorously evaluates reasoning abilities. Overall, the proposed methods offer valuable insights for advancing abstract reasoning in LLMs.", "theoretical_claims": "The paper presents several theoretical claims supported by formal proofs, including Theorem 3.7 (validity of Γ for Rule-Given potential), Theorem 3.8 (validity of ∆ for Rule-Inductive abstraction), and Theorem 3.9 (score range interpretation). These proofs aim to establish the mathematical soundness of the proposed metrics. The proofs appear to be correctly formulated and logically consistent, providing a rigorous foundation for the metrics. The Law of Large Numbers is appropriately invoked in Theorem 3.7, and the invariance properties are well-argued in Theorem 3.8. The score interpretation in Theorem 3.9 is also logically structured.", "experimental_designs_or_analyses": "The experimental designs and analyses in the paper are sound and valid. The authors evaluated various LLMs using a comprehensive benchmark with both Direct Prompting and Chain-of-Thought strategies. The results reveal some limitations in abstract reasoning and memory dependence. The findings may be helpful for improvement in LLMs.", "relation_to_broader_scientific_literature": "The paper integrates cognitive science theories of abstraction and reasoning into a novel framework, extending prior work on generalization and reasoning evaluation. Its metrics and benchmark design address some limitations of existing methods, focusing on symbolic tasks and diverse reasoning facets. Empirical findings align with known challenges in LLMs' generalization and highlight the need for improved reasoning capabilities, advancing the discourse on AI abstract reasoning.", "essential_references_not_discussed": "The paper provides a good review of related work in abstract reasoning and evaluation of large language models (LLMs). However, there are a few works that could further enrich the context and provide additional depth to the discussion:\n\n1. The integration of neural networks with symbolic reasoning is an active area of research. The paper could reference Neuro-Symbolic AI: The Third Wave by [Artur d'Avila Garcez, 2020], which discusses the combination of neural and symbolic approaches to improve reasoning capabilities. \n\n2. The paper's focus on benchmarking abstract reasoning could be complemented by citing classic cognitive science benchmarks. For example, the Raven's Progressive Matrices have long been used to evaluate human abstract reasoning. Including such benchmarks would provide a broader perspective on the challenges and methods used in evaluating reasoning. (Using the text-based RPM problem in [Taylor Webb, Nature Human Behaviour, 2023])\n\n3. This paper could cite \"The Measure of Intelligence by [Chollet'19]\", which provides a comprehensive framework for evaluating intelligence in AI systems. This work offers valuable context for the theoretical underpinnings of the paper's approach to abstract reasoning.", "other_strengths_and_weaknesses": "+:\n\n1. The benchmark is designed to cover a wide range of tasks. This allows for a good view of different facets of abstract reasoning.\n\n2. This paper introduces both \"Abstract Reasoning Score\" and \"Memory Dependence Score\" to provide a nuanced evaluation of model performance\n\n3. The paper provides a strong theoretical foundation for abstract reasoning by formalizing the processes of abstraction and reasoning. \n\n4. The paper includes extensive evaluations across various LLMs (7B-70B scale, API-based models, and agent frameworks).\n\n-:\n\n1. The paper acknowledges the influence of training data on model performance but does not provide a detailed analysis of how different training data configurations might affect abstract reasoning capabilities. \n\n2. The paper does not provide a direct comparison of LLM performance to human-level reasoning. \n\n3. The paper could provide more detailed implementation details for the benchmark tasks, such as specific examples of symbol remapping and task configurations.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See Strengths And Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9CB0sQi8zt", "reviewer": "Reviewer_CDZ6", "summary": "This work build a benchmark of arithmetic computation tasks that targets at the abstract reasoning abilities of large language models, and finds out that the power existing large language models relies on the task domains.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Concrete Instances vs. Abstract Features\n\nOur paper distinguishes concrete instances (C)—detailed input strings containing surface-level information—from abstract features (A), which capture only the essential properties required for reasoning. For example, a concrete description of a dog may include breed and color, whereas its abstract feature reduces these details to “dog.” Our mapping f: C->A is designed to force models to look beyond token memorization; an inverse mapping (from A->C) is unnecessary for our task.\n\n---\n\n## 2. Notation and Example in Definition 3.2\n\n### (a) Notation \"$A \\times R$\"\n\nIn Definition 3.2, the notation $A \\times R$ indicates the set of all ordered pairs `(a, r)`, where each `a` belongs to the abstract feature set `A` (Section 3.1) and each `r` belongs to the rule set `R` (Section 3.2). The reasoning function `Re` is defined as\nRe: $A \\times R$→Q,\nwhich means that for any input pair `(a, r)`, the function produces a corresponding conclusion `q` in `Q`. This formulation follows standard mathematical conventions (e.g., see [Cormen et al., 2009]) for combining elements from two previously defined sets.  \n\n### (b) The Example Involving Uncertainty\n\nThe phrase “This dog is likely to bark” illustrates that typical human reasoning can involve uncertainty. However, in strictly symbolic or arithmetic tasks, the reasoning outcome is deterministic. The example is meant only to show how abstract features and rules combine, with the particular uncertainty reflecting context rather than a general property of all reasoning functions.\n\n---\n\n## 3. Theoretical Foundation of Theorem 3.7\n\nTheorem 3.7 states that if a large language model achieves an abstract reasoning score Γ on our test set T that meets or exceeds a threshold γ, then for every task (c, r, q) in T the probability of correctly outputting q (via the abstraction mapping f and the reasoning function Re) is at least γ. In essence, a high Γ confirms the model’s robust ability to perform abstract reasoning across T.\n\nThis theorem is grounded in our rigorous definitions of f and Re, which provide a clear correspondence among the inputs, the applied rules, and the correct outputs. Although the main text offers only a brief proof, more detailed proofs and explanations are available in the Appendix.\n\n---\n\n## 4. Score Range Interpretation (Theorem 3.9)\n\nFor Theorem 3.9, we define the combined score as  \nF(Γ, Δ) = w₁·Γ + w₂·(1 − Δ)\nand, according to Theorem 3.8, a higher threshold γ implies a higher Γ. The weights w1 and w2 are chosen based on task requirements. Even when w1 = w2, the two components remain distinct: Γ measures the reasoning accuracy, while (1 − Δ) reflects the model's robustness against dependence on specific tokens. Thus, the score range interpretation stays relevant as each component captures a distinct aspect of performance.\n\n---\n\n## Additional Responses\n\n### 1. On Supporting Abstract Reasoning Abilities\n\nOur task design is not limited to simple arithmetic operations. Instead, we adopt a symbol remapping strategy that forces models to abandon reliance on surface-level patterns and requires them to extract and apply the underlying abstract rules. For example, in a date calculation task (e.g., `days between dates([2024, 07, 29], [2021, 10, 31]) = ?`), the model is required not only to perform numerical computations but also to understand the intrinsic logic between time and dates, which manifests a higher demand for abstract reasoning. Even large-scale models did not achieve above 60% accuracy on these tasks, which shows that our benchmark is sufficiently challenging for current models’ abstract reasoning abilities.\n\n### 2. On the Conciseness of Theorems and Theoretical Framework\n\nWe acknowledge that our current exposition of definitions and theorems is brief. The definitions of the abstraction mapping (f) and the reasoning function (Re) serve as the foundational building blocks of our mathematical model, and the corresponding theorems (e.g., Theorem 3.7 and Theorem 3.9) follow standard analytical approaches. Detailed proofs and clarifications are provided in the Appendix.\n\n### 3. On the Benchmark’s Focus on Arithmetic Tasks\n\nWhile our benchmark primarily features arithmetic and related symbolic tasks, it is not confined to simple calculations. In addition to basic arithmetic, our benchmark incorporates extended calculation tasks, operations in various number bases, and date calculations. Most tasks are designed using symbol remapping techniques to destabilize simple memorization, thereby revealing models’ inability to generalize abstract rules beyond familiar forms. \n\n---\n\nBecause your review was relatively brief, we hope you will raise more questions or concerns to help us further improve the quality of our paper. Your additional feedback will be invaluable as we continue to refine our theoretical framework and experimental design.\n\nThank you for your review.\n\nSincerely,  \nThe Author Team", "claims_and_evidence": "See below", "methods_and_evaluation_criteria": "The benchmark provides signal that demonstrates the limitation of the reasoning abilities of existing large language models, but not sufficient in supporting the abstract reasoning abilities.", "theoretical_claims": "The theorems are very brief and not placed in a well-described framework with all the concepts well explained.", "experimental_designs_or_analyses": "The benchmark consists of basic arithmetic tasks, where the areas where abstract reasoning are more broadly discussed (e.g. more complex logical reasoning, commonsense reasoning) is not discussed.", "relation_to_broader_scientific_literature": "The topic would raise interest to the broader scope of audience.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "There are several places worth mentioning in Section 3.\n- For Definition 3.1, the key difference between concrete instances and abstract features is not explained. In other words, what makes a string concrete instances instead of abstract features. What is the difference between the mapping C->A and A->C?\n- For Definition 3.2, the 'x' operation in 'AxR' is not explained. Meanwhile, the example is confusing as the conclusion applied here is 'This dog is likely to bark' without introducing the origin of the uncertainty here. Does this also apply to all the reasoning functions?\n- Theorem 3.7 is hard to understand, the foundation of the validity is not explained.\n- For theorem 3.9, the score range interpretation can be irrelevant to \\gamma is w1=w2 according to the definition.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TWV2fxcSsO", "reviewer": "Reviewer_SZxB", "summary": "This paper introduces a rigorously designed and theoretically grounded benchmark for a crucial yet underexplored area: the evaluation of abstract reasoning in Large Language Models (LLMs). The work establishes a clear mathematical framework that defines abstract reasoning as the ability to extract invariant patterns and consistently apply rules, independent of superficial representations. Central to the contribution are two novel, complementary metrics: the Abstract Reasoning Score (Γ) to quantify baseline accuracy and the Memory Dependence Score (Δ), a diagnostic metric innovatively designed to reveal the degree to which models rely on memorization versus genuine abstraction. Through systematic symbol remapping in carefully designed rule-based tasks, the benchmark compels models to demonstrate true pattern recognition. Extensive empirical evaluations across a diverse suite of LLMs, encompassing varying scales and prompting strategies, convincingly demonstrate the limitations of current models in abstract reasoning, particularly in symbolic manipulation and generalization. The paper's findings powerfully advocate for the use of this benchmark to guide future progress in AI systems capable of robust abstract thought, representing a significant step forward in LLM evaluation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1. Task Complexity Expansion\n\nWe agree that further expanding task complexity is essential. In future work, we will:\n\n- Introduce multi-step reasoning tasks that require chaining hypotheses, intermediate conclusions, and final integration.\n- Incorporate hierarchical rules with nested, conditional, and conflict resolution components.\n- Implement dynamic difficulty adjustment using parameters (e.g., reasoning steps, rule density, interference factors) to generate tasks that evolve with model proficiency.\n---\n### 2. Enhancing Rule-Inductive Reasoning\n\nWe plan to deepen our investigation into rule induction by:\n\n- Increasing test difficulty—e.g., training with 8-bit binary data and testing with 16-bit numbers—to challenge the model’s generalization.\n- Recording and validating the reasoning steps, not just the final answer; automated tools will assess the plausibility of the reasoning chain.\n- Including detailed error case analyses, primarily in the Symbolic Math Abstraction (SMA) and Symbolic Reasoning (SR) categories, to illuminate common failure modes.\n---\n### 3. Benchmark Evolution and Maintenance\n\nWe have a three-stage plan to keep the benchmark state‑of‑the‑art:\n\n- **Short-term:** Open-source the evaluation framework with detailed examples and error analysis.\n- **Mid-term:** Develop a dynamic task difficulty system through parameterization of reasoning steps and rule complexity.\n- **Long-term:** Build an adaptive challenge mechanism that auto-generates advanced tasks as models improve. We will also create robust feedback channels and community platforms to encourage continuous improvement.\n---\n### 4. Detailed Examples and Error Analysis for Rule Induction\n\nWe will add:\n\n- Inclusion of Detailed Case Studies: We will include a dedicated section in our open-source GitHub repository that presents detailed case studies, particularly focusing on SMA and SR tasks.\n- Error Pattern Analysis: Along with concrete examples, we will provide analyses of common error patterns, explaining the underlying causes and typical failure modes in rule-induced tasks.\n  We thank you for emphasizing the need for more specific examples and error analyses. We agree that these enhancements will provide immediate, valuable insights into model limitations and help guide future improvements.\n---\n### 5. Dataset Openness and Community Engagement\n\nTo maximize community impact, we will:\n\n- Open-source the complete project on GitHub, including data generation code, symbol mapping, the full dataset, and all evaluation scripts.\n- Provide thorough documentation and user guides.\n\n---\nWe appreciate your detailed feedback. Your recommendations have pinpointed vital areas for improvement. In response, we have developed clear and actionable measures to refine task complexity, enhance rule induction analysis, and ensure the benchmark remains dynamic and accessible. These improvements will help us advance robust abstract reasoning evaluation in LLMs.\n\nSincerely,  \nThe Author Team", "claims_and_evidence": "The paper’s core claim, asserting that contemporary LLMs exhibit significant deficits in abstract reasoning and are demonstrably prone to memorization—especially when confronted with novel symbolic representations—is convincingly and thoroughly substantiated. The experimental results are particularly compelling: the stark performance decline observed in Number Base Reasoning (NBR) tasks, even for the largest models, alongside the insightful diagnostic capabilities of the Memory Dependence Score (Δ), provide robust empirical support. For example, the consistent increase in Δ observed across models highlights a clear reliance on specific symbol tokens. The benchmark's innovative design, based on systematic symbol remapping within rule-based tasks, is demonstrably effective in isolating and measuring abstract reasoning, moving beyond superficial pattern matching or simple statistical correlations. The comprehensive evaluations, spanning a wide array of models, from 7B to 70B scale, and diverse prompting techniques, further reinforce the strength of the evidence presented. The paper effectively and persuasively argues against the sufficiency of relying solely on standard accuracy metrics for a nuanced assessment of genuine reasoning capacity in LLMs.", "methods_and_evaluation_criteria": "The methodology employed in this work stands as a significant strength. The benchmark design itself, particularly the ingenious incorporation of systematic symbol remapping, represents a valuable methodological contribution to the field of LLM evaluation. The proposed metrics, Γ and Δ, offer a well-defined and crucially, complementary framework for rigorous evaluation. The Memory Dependence Score (Δ) is particularly noteworthy as it provides a diagnostic measure capable of differentiating between genuine abstract reasoning and mere memorization – a vital distinction frequently overlooked in many LLM evaluations. The deliberate use of rule-based tasks, thoughtfully categorized into Basic Computation (BC), Extended Calculation (EC), Number Base Reasoning (NBR), Math Application (MA), Symbolic Math Abstraction (SMA), and Symbolic Reasoning (SR), provides a structured and scalable approach to evaluating diverse facets of abstract reasoning. The experimental setup, meticulously encompassing a diverse range of models, prompting strategies (including both Direct Prompting and Chain-of-Thought), and computational resources, demonstrates appropriate rigor and careful consideration of experimental controls.", "theoretical_claims": "The paper's rigorously developed theoretical framework represents a substantial strength and a defining characteristic of this work. It provides precise formal definitions for key concepts including abstraction mapping, reasoning functions, and composite reasoning functions, offering a solid and mathematically sound conceptual grounding for the benchmark. The theorems presented (3.7, 3.8, 3.9) are not merely assertions but are theoretically validated, demonstrating the validity of Γ for assessing Rule-Given potential and, importantly, Δ as a robust measure for evaluating Rule-Inductive Abstraction capabilities. This level of theoretical rigor significantly distinguishes the paper from purely empirical evaluations and provides a robust foundation for both the empirical findings and the proposed evaluation methodology, enhancing the paper’s overall impact and credibility.", "experimental_designs_or_analyses": "The experimental design is not only well-executed but also remarkably comprehensive in its scope and depth. The authors have systematically and meticulously evaluated a diverse range of LLMs, thoughtfully spanning different model scales (from 7B to 70B), varying architectures, and distinct access methods (encompassing both open-source and API-based models). The deliberate inclusion of different prompting strategies, namely Direct Prompting and Chain-of-Thought (CoT), allows for a nuanced and granular analysis of model behavior and reasoning capabilities under varying conditions and input formats. The detailed and transparent reporting of results across all defined task categories, and particularly the fine-grained analysis of the Memory Dependence Score (Δ) at the level of both operand and operator symbols, provides exceptionally valuable and granular insights into the inherent limitations of current LLMs in abstract reasoning. The automated evaluation process, which incorporates gpt-40-mini for parsing responses and ensuring objective answer correctness assessment, further enhances the objectivity, reliability, and overall rigor of the study.", "relation_to_broader_scientific_literature": "This paper is exceptionally well-contextualized within the broader and highly relevant scientific literature concerning abstract reasoning in both cognitive science and artificial intelligence. The related work section is particularly effective in its detailed discussion of existing benchmarks designed to assess AI reasoning, clearly articulating their limitations in the context of rigorously evaluating abstract reasoning within LLMs, and, crucially, in effectively differentiating between genuine abstraction and superficial memorization. The paper thoughtfully builds upon established theoretical perspectives from cognitive science and clearly and persuasively articulates the specific advancements and novel contributions offered by the proposed benchmark and evaluation metrics. The grounding of the work in established cognitive science literature, particularly regarding the distinction between Rule-Given and Rule-Inductive reasoning, is entirely appropriate and significantly strengthens the paper's scholarly context and theoretical underpinnings.", "essential_references_not_discussed": "While the literature review is demonstrably thorough and highly relevant to contemporary LLM research, one could argue for the inclusion of more foundational key works from the pre-LLM era in the field of symbolic AI and rule-based reasoning. For instance, seminal works on production systems, knowledge representation, and early symbolic reasoning architectures, while not strictly essential for grasping the paper’s core contributions in the context of modern LLM evaluation, could provide a more complete and historically nuanced perspective on the evolution of AI reasoning research. However, the current literature review is highly focused and effectively addresses the most pertinent prior work directly relevant to the paper's contributions within the current landscape of LLM research.", "other_strengths_and_weaknesses": "Strengths:\n\n- Novel and Theoretically Grounded Benchmark: The paper introduces a genuinely novel and exceptionally well-theoretically motivated benchmark that directly addresses a critical gap in the rigorous evaluation of abstract reasoning capabilities in LLMs.\n\n- Diagnostic Memory Dependence Score (Δ): The innovative introduction of the Memory Dependence Score (Δ) stands out as a significant strength, providing a uniquely valuable and diagnostic tool for disentangling genuine abstraction from mere memorization.\n\n- Comprehensive and Rigorous Empirical Evaluation: The empirical evaluation is remarkably comprehensive, covering a wide and diverse range of models and prompting strategies, providing exceptionally robust and generalizable results.\n\n- Clear Identification of Critical Research Gaps: The paper effectively and persuasively identifies fundamental limitations in current LLMs' abstract reasoning capabilities and clearly points towards valuable and impactful future research directions, advancing the field.\n\n- Well-Defined Task Categories and Dataset: The structured categorization of tasks (BC, EC, NBR, MA, SMA, SR) and the creation of a well-documented dataset contribute significantly to the reproducibility and usability of the benchmark.\n\nWeaknesses:\n\n- Potential for Further Task Complexity: While already comprehensive, the scope of the benchmark tasks could be further expanded in future iterations to incorporate even more complex and nuanced reasoning scenarios, particularly those involving multi-step inference and more intricate rule structures, to maintain its leading-edge utility as LLMs continue to advance.\n\n- Depth of Rule-Inductive Reasoning Exploration: While the benchmark addresses rule-inductive reasoning, further and deeper exploration of model performance specifically in the domain of rule induction from genuinely novel data – data designed to minimize any reliance on pre-existing patterns or biases learned during pre-training \n\n– could potentially provide even richer and more granular insights into this challenging aspect of abstract reasoning for LLMs.", "other_comments_or_suggestions": "- Future Benchmark Development and Maintenance: To maximize long-term impact, consider explicitly emphasizing the ongoing development, community engagement, and potential expansions of the benchmark in future publications and releases. Establishing a clear roadmap for incorporating increasingly complex reasoning tasks and adapting to advancements in LLM technology will be crucial for maintaining its continued relevance and utility to the research community.\n\n- Elaborate on Rule-Inductive Reasoning Challenges: Further elaboration, perhaps incorporating more concrete and illustrative examples, specifically detailing the nuanced challenges LLMs demonstrably face when engaging in rule-inductive reasoning tasks, would be highly beneficial for readers less familiar with the intricacies of this cognitive process.\n\n- Dataset Accessibility, Documentation, and Community Building: Ensuring open and easily accessible access to the meticulously curated dataset, coupled with providing exceptionally thorough and user-friendly documentation of its structure, generation process, and intended usage, is absolutely crucial for maximizing reproducibility, fostering broader community adoption, and ensuring the long-term impact of this valuable benchmark. Consider creating a dedicated project website or repository to facilitate access and community contributions.", "questions_for_authors": "- Benchmark Evolution Roadmap: Could the authors outline their specific plans and roadmap for the continued evolution and expansion of the benchmark? Specifically, what types of increasingly complex reasoning tasks are envisioned for future iterations, and how will the benchmark be adapted to remain effective and challenging as LLMs continue to advance in capabilities? Understanding this long-term vision would further solidify the benchmark's value as a lasting contribution.\n\n- Deep Dive into Rule Induction Failure Modes: To enhance practical understanding for researchers seeking to improve LLM rule induction, could the authors provide more specific, illustrative examples and a more detailed analysis of the typical failure modes observed in LLMs when tackling rule induction tasks? Focusing on concrete examples, particularly from the SMA and SR task categories, would offer valuable insights into the precise nature of the limitations and guide targeted improvement efforts.\n\n- Dataset Access and Community Engagement Strategy: What is the authors' strategy for ensuring broad and sustainable access to the benchmark dataset and fostering community engagement around its use and potential expansion? Clearly articulating plans for dataset accessibility, comprehensive documentation, and community contribution mechanisms would maximize the impact and utility of this valuable resource for the wider AI research community.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a formal framework for studying consistent abstract reasoning, and applies it to evaluate a variety of LLMs.
After discussion, the majority of the reviewers generally agree that the paper offers a useful contribution to the literature, though various additions such as human baseline evaluations would make it a stronger and more well-rounded paper. On the whole, this work seems to offer some useful analysis, and is reasonably clearly presented.
Decision: Accept (poster)