Language Models are Symbolic Learners in Arithmetic
Authors: Chunyuan Deng
Zhiqi Li
Roy Xie
Ruidi Chang
Hanjie Chen
Venue: Accepted by TMLR ()

Abstract: The prevailing question in LM performing arithmetic is whether these models learn to truly compute or if they simply master superficial pattern matching. In this paper, we argues for the latter, presenting evidence that LMs act as greedy symbolic learners, prioritizing the simplest possible shortcuts to fit the stats of dataset to solve arithmetic tasks. To investigate this, we introduce \textbf{subgroup induction}, a practical framework adapted from Solomonoff Induction (SI), one of the most powerful universal predictors. Our framework analyzes arithmetic problems by breaking them down into ``subgroups''—minimal mappings between a few input digits and a single output digit. Our primary metric, subgroup quality, measures the viability of these shortcuts. Experiments reveal a distinct U-shaped accuracy pattern in multi-digit multiplication: LMs quickly master the first and last output digits while struggling with those in the middle. We demonstrate this U-shape is not coincidental; it perfectly mirrors the quality of the simplest possible subgroups, those requiring the fewest input tokens. This alignment suggests a core learning mechanism: LMs first learn easy, low-token shortcuts and only incorporate more complex, multi-token patterns as training progresses. They do not learn the algorithm of multiplication but rather a hierarchy of increasingly complex symbol-to-symbol mappings. Ultimately, our findings suggest that the path to arithmetic mastery for LMs is not paved with algorithms, but with a cascade of simple, hierarchically-learned symbolic shortcuts. The code is at https://github.com/chili-lab/Symbolic-Arithmetic.

Published in Transactions on Machine Learning Research (01/2026)
Language Models are Symbolic Learners in Arithmetic
Chunyuan Deng
chamsuan. dww.deennggnreeceeoc
Deportment of Computer Science
Rice University
Zhiqi Li
=k3167@gotech. edu
College Computing
Georgia Institute of Technology
Roy Xie
ruogu.rie@duke.co edu
Deportment of Computer Science Duke Universily
Ruidi Chang
re151 Orice. edu
Department of Computer Science
Rice University
Hanjie Chen
hanjie Orice. edu
Department of Computer Science and Ken Kennedy Institute
Rice University
Reviewed on OpenReview; https: // oppnreveew... net/ pet/forum? id- QSb PPg1aUM
Abstract
The prevailing question in LM performing arithmetic is whether these models learn to truly compute or if they simply master superficial pattern matching. In this paper, we argues for the latter, presenting evidence that LMs act as greedy symbolic learners, prioritizing the simplest possible shortcuts to the stats of dataset to solve arithmetic tasks. To investigate this, we introduce subgroup induction, practical framework adapted from Solomonoff Induction (SI), one of the most powerful universal predictors. Our framework analyzes arithmetic problems by breaking them down into minigal mappings between few input digits and single output digit. Our primary metrie, subgroup quality, measures the viability of these shorteuts. Experiments reveal distinct U-shaped accuracy pattern in multi-digit multiplication: LMS quickly master the first and last output digits while struggling with those in the middle. We demonstrate this U-shape is not coincidental; it perfectly mirrors the quality of the simplest possible subgroups, those requiring the fewest input tokens. This alignment suggests a core learning mechanism: LMs first learn easy, low-token shortcuts and only incorporate more complex, multi-token patterns training progresses. They do not learn the algorithm of multiplication but rather hierarchy of increasingly complex symbol- toosymbol mappings. Ultimately, our findings suggest that the path to arithmetic mastery for LMs is not paved with algorithms, but with cascade of simple, hierarchically-leareed symbolic shortcuts. The code is at httpss//githubbcom http://hww.ooo//illlla////////// Arithmetic.
Introduction
The saturation of modern math benchmarks like GSM8K Cobbe et al. 2021 by frontier language models such as GPT-4o (OpenAI al.) (024) and Claude (nthropicc 2024) marks a significant milestone AI. As the research community pivots to grander challenges like Olympic-leve mathematies, this success raises a more fundamental question: are these models developing genuine numerical reasoning, they exhibiting

---

Published in Transactions on Machine Learning Research (01/2026)
Solomonoff Induction
Subgroup Induction
Universal Turing Machine (UTM)
Language Model (LM)
Subgroup s
A1
A2
B1
B2
Subgroup s2
3
2
6
9
5
2
x
Mitsug AgMr(0 M(X.r)
Bayesian U Updating
Mo Xxx)
Posterial preciation given previcus data with hypothess.
C1
C2
C3
C4
((p): phopremm (hypo thes length defined Koinoglor complexity.
Subgroup Quality ((s) (A1A2 C1)
Subgroup Complexity
Kolmoiglov Complexity
Subgroup Entropy H(L)
(X)
Σ
2 Upp)
Occam's 's Razor
83
(p: -xx
Assign the shortest program higher probobilty
To predict CZ, LMs need 54?
Subgroup Selection
Overview of Subgroup Induction. Solomonoff Induction (ST) is conceptual framework that utilizes a universal predictor, such as UTMs, to make predictions. Inspired by SI's principle of Occam's 's Razor, subgroup induction a pragmatic framework designed to uncover the shortcut-secking mechanisms LMs use to perform arithmetic.
Figure 1:
an elaborate illusion of it, woven from the statistical patterns of their vast training data? In short, do they learn to truly compule, or do they simply master sophisticated pattern matching?
In this paper, we argue for the latter. We present compelling evidence that LMs act as groody symbolic learners, prioritizing the simplest possible "shortcuts" solve arithmetic tasks. Understanding this distinction is critical, as has profound implications for model reliability, out-of-distributiin generalization, and the future of trustworthy Al reasoning. To probe this behavior, use arithmetic as controlled laboratory. Unlike open-ended reasoning, arithmetic tasks are close-formed the underlying data generation function I is pre-defined and immutable. This provides a gold-standard ground truth against which we can rigorously analyze the learning strategies LMs actually develop.
To formalize this investigation, we introduce a practical framework called subgroup induction (sce Figure Our approach is inspired by one of the oldest principles in science and learning theory: Occam's Razor, which suggests a preference for simpler explanations. This principle is formally embodicd in thcoretical frameworks like Solomonoff Induction (SI). We adapt this core idea. into a concrete, testable hypothesis for LMs: a "simpler" solution for predicting an output digit is one that relies on the simplest possible input pattern-thae is, a minimal set of input digits (tokens).
Within this framework, a subgroup becomes our basic unit of analysis. It represents a potential shortcut, defined as a minimal mapping from a subset of input digits to a single output digit. For example, does an LM only need to see the last digits '..3" and '..4' to learn that the product must end in s..2"? To measure the viability of such shortcuts, introduce two complementary measures of subgroup complexity. Our primary metric, subgroup qualily, quantifies the reliability of a given shorteut across the entire data distribution. A second metric, subgroup entropy, measures the inherent ambiguity or difficulty of the prediction at each output position, serving as a robust tool for error estimation.
Our experiments reveal a distinct and consistent U-shaped accuracy pattern in multi-digit multiplication: LMs rapidly master the first and last output digits but consistently struggle with those the middle. We demonstrate this U-shape is not coincidental but is the "smoking gun" for our shortcut hypothesis. The model's performance curve perfectly mirrors the quality of the simplest possible subgroups those requiring the fewest input tokens. This powerful alignment reveals core learning mechanism: LMs instinctively learn easy, low-token shorteuts first and only incorporate more compler, multi-token patterns when forced to by the learning objective. They do not. learn the abstract algorithm of multiplication, instead, they discover a hierarchy of symbol-to-ss to-symbol mappings, starting with the path of least resistance.
Furthermore, we show that subgroup entropy powerful tool for analyzing more complex reasoning. In a Chain-of-Thought (CoT) setting (Wei et al. 2023), different reasoning paths be viewed decompositions
2

---

Published in Transactions on Machine Learning Research (01/2026)
of a problem into simpler steps. Our framework quantifies the 'symbolic complexity" of each path, and we find that the CoT path with the lowest aggregate entropy consistently achieves the best performance. This suggests LMs prefer reasoning strategies composed of the easiest possible sequence of symbolic shortcuts.
1 Preliminaries
In this section, we introduce the preliminarics the universal Turing machine and Solomonoff induction. We then discuss how arithmetic learning can be incorporated into the SI framework, laying the groundwork for presenting our subgroup induction framework.
1.1 Solomonoff Induction
Solomonoff induction formalizes optimal prediction by combining computability and Bayesian principles. Tt. considers all possible hypotheses (as programs) weighted by two criteria: simplicity (shorter programs receive exponentially higher weight) and consistency with observed data.
The framework unifies three elements: a universal prior M over hypotheses, Kolmogorov complexity as a measure of simplicity, and Bayesian updating across the hypothesis space. This yields provably optimal predictions for sequences generated by any computable process. We formalize these components as follows:
Definition 1.1 (Universal Probability). The universal probability is a mixture all lower semicomputable semimeasures μ4, weighted by their algorithmic complexity:
M(x)=
Σ
2 2-((p)
p=t(p)=r
where l(p) is the length of program p on the reference UTM U. Each program p represents a hypothesis about the data-generating process.
Definition 1.2 (Occam's Razor). The principle of Occam's Razor is formalized through Kolmogorov complexity Ku (x), which is the length of the shortest program that outputs
Kv(x) min(!(p) U(p) x}
This assigns higher prior probabilities (2 (2-Ko(a)) to simpler hypotheses, as shorter programs have higher weights the universal probability.
Definition 1.3 (Bayesian Updating). Each program p maps a hypothesis h hypothesis space H. Given prior probability M and observation æıa, the posterior probability computed via Bayes' rule:
M(h(raa ((z1a)))M(h)
M(
The predictor updates across all hypotheses (programs):
M( (214)) = Σ M(zz+[
bER
1.2 Arithmetic Learning as Bridge to Practice
Arithmetic learning offers an ideal setting for studying inductive inference, as it involves ground-truth aata-generating functions. We will first provide a formal definition of arithmetic learning and then discuss the similarities that motivate the development a practical framework.
Arithmetic Learning Task. The task of arithmetic learning learn approximation to ground truth data generation function f that generalizes to unseen inputs. The ground truth N N + N maps input pairs to their arithmetic result:
f(a = a a b
where represents arithmetic operation like addition multiplication.
3

---

Published in Transactions on Machine Learning Research (01/2026)
Arithmetic Data Distribution. Consider training set Durain {(a!k) bi6), e(())) J [aa!), ,(()) for a binary operator f()). For n-digit arithmetic, inputs ((k) and ((8) can be viewed realizations of random variable sequences {A and {Br where each A, Br ~ ~0, 1..., 9), U is the uniform distribution. Similarly, cla) corresponds to sequence {Ci)-= with joint distribution:
/== where c(
P( (B.).)) = ( P(A))((
where [4) is the indicator function.
LMs as Predictive Distributions. LMs implement a distribution p0(2x+1121a) over next tokens, parame- terized by weights For arithmetic learning:
b) 2 (( b)
This approximates the true conditional probability of the arithmetic result.
Proposition 1.4 (Parallel Learning Frameworks). The theoretical SI framework and practical LM imple- mentation approach arithmetic learning through parallel mechanisms: 1. ST via Bayesion updating over programs:
Mícla, D)
Σ
M(clh)M((lD))
DCH
2. LMs via gradient descent on negative log- loglikklihoodd:
L(0) =
Σ (a,b,c)ED
(claa b)
Both frameworks exhibit convergence behaviors on arithmetic tasks: SI identifies the shortest program computing with probability 1 as [D] + 2. LMs learn approximate implementation minimizing empirical risk, bounded by model capacity and optimization dynamics.
These comparisons demonstrate that it is possible to use the core principles of Solomonoff induction to understand the behavior of language models in arithmetic tasks. This insight motivates us to develop a pragmatic framework capable of providing computable properties to analyze LMs' learning mechanisms in arithmetic.
2
Subgroup Induction
In this section, we introduce the subgroup induction framework. To constrain the hypothesis space, we define a computable property by specifying the concept of subgroup.
2.1 Subgroup
Definition 2.1 (Subgroup). For n-digit arithmetic f(a,b) b) = c, the subgroup s is defined as (Ai, Bi, and C₁ have the same meaning as in isubsection 1.2):
s E S = ((AAB), C) A (AHE, (B.) Ce (C, C2....,Cml
where A and I are subsets of the input variable sequence, C represents a single variable in the output sequence, n and m is the length for input and output number. For convenience, we represent A, B, C by the sequence their elements order. For example, A = (Ai, A3, As}, we write A as A, AgAs, Examples for 2-digits multiplication A₁ B, C3C.C
(I),, C1)
((A2,B)) C4)
((A142, C1)
4

---

Published in Transactions on Machine Learning Research (01/2026)
((A1,BrB2), Ca) ((A1A2, By B2), C4)
In these subgroups, & and are subsels of impul variable sequences, while C is restricted to single voriable. This design aligns with the next-token prediction paradigm in language modeling!
Computability. By using subgroup decomposition, arithmetic tasks with n-digit input and m-digits output can be divided into (2 - 1) x 2m subgroups overall Essentially, treat the LMs' choice of using symbol information, which corresponds to choosing subgroup, hypothesis
2.2 Subgroup Complexity
In Solomonoff Induction, hypotheses or universal turing machine programs are assigned complexity measure. Kolmogorov complexity, a key component of this framework, embodies Occam's Razor by assigning higher probability to shorter programs. In our framework, we treat subgroup usage as a hypothesis. Thus, key challenge is defining an appropriate mcasure for subgroup complexity.
Intuition. In multiplication, the leftmost input digits largely determine the first output digits, hinting that predicting some positions might be easy. Also, predicting a single value 0 is easier than predicting a range 0-9, showing that output voriability matters. These two obscrvations motivate our two proposed complexity measures respectively.
Definition 2.2 (Subgroup Quality). Given original dataset D, for any subgroup s (), E Sn, the subgroup quality for multiplication is:
(s) Ba [[ ((((a,A) x o(b d(c, 2))l
(1)
where I(-) is an indicator function returns 1 if argument is true, and 0 otherwise. & is masking function to set digits at positions of variables not in subgroup to 0 (see Algorithm Extending this definition from multiplication to any operator f()), we could generalized version:
((s) = Baa [F(o(a, ((b, = dlc [
(2)
where f(+)) any binary operator for the arithmetic tasks.
Definition 2.3 (Subgroup Entropy). The subgroup space is defined based on the label space for each subgroup. Given s, the label space is given by:
L, fc ||cPPCC = c) > 0}
For given subgroup, the label space consists of all labels observed at its output positions in the dataset. The subgroup entropy is then defined as:
H(s) = Σ P(C = c) loga P(C =c)
(3)
DEL.
2.3 Discussions
For given subgroup 8, subgroup complexity is defined using two computable measures:
TTo prevent tokenization issues, add space between each digit
that each digit is tokenized separately.
IIput and output each have 2n digits. Considering all possible permutations for input (excluding the empty set), have (22n 1) combinations. Therefore, the total number of subgroups (22n 1) x 2m.
MM. models like linear linear/logistll regression or simple MLP, the hypothesis is characterized by the model weights (porameters) that define the mapping function me. But this also face the challenge of incomputability due to the infinite hypothesss space.
5

---

Published in Transactions on Machine Learning Research (01/2026)
Algorithm Subgroup Quality for n-digit Multiplication
Require: Original dataset Prain of size N, set of all subgroups
Note: MASK sets digits at positions of variables not in the subgroup to 0
1: for each subgroup
((A, B), C) E S do
2: correct + 0
3: for each (a, b, c) E Draain do
4: Amasked f- MASK(a, (a, A)
Dmaaked - MASK (b, B)
Cmasked MASK(C, C)
pred & Umasked Umasked
if pred = Cmasked then correct + correct + 1
10:
end if end for
11:
12:
((s) + correct N
subgroup quality
13:
end for
Qs): subgroup quality that measures how accurate the mapping is from the input positions to the output position.
(s): Subgroup entropy that measures the uncertainty or variability within the subgroup's 's output labels.
These two measures are proposed with different motivations. Next, we will explore the effectiveness of these two measures in quantifying LMs' learning behavior.
3 Arithmetic Learning in Language Models
We investigate L.Ms' learning mechanism using intuitive yet straightforward approach by observing the relationship between subgroup complexity and position-level accuracy.
3.1 Experiment Settings
We trained Gemma-2-2B TTea et al. 2024) and Llama-3.1-8B (Dubey el al. 2024) on multiplication arithmetic tasks using datasets of varying sizes: 6.48K, 12.96K, 32.4K, and 64.8K examples. Our experiments covered multiplication of to 5-digit numbers, resulting in outputs ranging from 10 digits. For simplicity and to ensure focused comparison, we did not optimize hyperparameters for either Gemma or Llama models, preliminary experiments indicated robust performance with default settings. Detailed experimental setup information can be found in Appendix A
3.2 Position-level Accuracy are U-shaped
We compute the accuracy at each position in the output sequence. Figure 2 reveals a phenomenon overlooked in previous studies that fundamentally challenges our understanding of how language models learn arithmetic operations. Contrary to the common assumption that position-level accuracy docrcases monotonically from right to left due to carryover effects and least-to-most significant digit calculations (Lee al. 2023; Zhang- Li et al. 2024), results show U-shaped accuracy curve in both Cemma-2-2BB and Llama-3 3. across all multiplication complexities.
3.1-BB 3.1-8-
models
This U-shaped pattern exhibits several noteworthy characteristics. Accuracy peaks at the beginning and end positions, consistently exceeding 95% regardless of training set size, while dramatically dropping to approximately 10% in the middle positions, particularly pronounced higher-digit multiplication tasks. The consistency of this pattern across different model architectures (Gemma-2 2-2B and Llama-3.1-88 and training set sizes (ranging from 6.48K to 64.8K examples) suggests that this is not artifact of specific training

---

Published in Transactions on Machine Learning Research (01/2026)
bddgtt Mulippication Gemrna-2-08
4-diiits Multiplcction- Gemma-2-38 2-88
5-digtt Muttplication Gemman-230
100%
100%
Acunney (M)
Accurscy 1N6
Accuracy (S6)
60%
074
Tiaknng fe Sea
409
640
12.80c
20%
0%
0%
5
5
10
Position
Positio
Position
3:dgits Multipicationn Aamaa-3.1-8
4-digits Multiplication - Liama-3.1-88
5-dgits Muliificaioon. Llame-81148
100%
80%
65%
Accurasy (56)
60%
Accuracy
60%
Tramiy ug Sae
40%
40%
CAEK
20%
BK
2
9 10
Position
Position
Position
Figure 2: Position-level Accuracy from Gemma-2-2B and Llama-3.1-8B.
conditions but rather reflects a fundamental aspect of how transformer models process sequential arithmetic operations.
The emergence of this U-shaped accuracy distribution provides compelling evidence that the difficulty in learning mullippicaation is concentrated in the middle posilions rather than at the beginming or end. This finding has profound implications for our understanding of arithmetic learning in large language models. The high accuracy at terminal positions suggests that models can effectively learn to identify the correct starting and ending digits through pattern recognition, possibly by memorizing frequent digit combinations or leveraging positional embeddings. Meanwhile, the poor performance middle positions indicates that models struggle with the complex interdependencies and carry operations that oceur during the intermediate steps of multi-digit multiplication.
Furthermore, this pattern intensifies with increasing digit complexity, evidenced by the more pronounced accuracy drops in 5-digit multiplication compared to 3-digit tasks. The robustness of this phenomenon across different training set sizes also suggests that simply increasing the volume of training data may not be sufficient to overcome the inherent difficulty of middle-position calculations, pointing toward the need for more sophisticated training strategies or architectural modifications specifically designed to handle sequential arithmetic dependencies.
3.3 Why Subgroup Quality Q(s) Explains the U-Shaped Accuracy
What Q(s) measures. For output digit position Cj, subgroup ((A, B), Cj) keeps only a subset of input digits from A and B and masks the rest to zero. The subgroup quality Q(s) is the accuracy of predicting Cj using this masked product (Alg D Intuitively, Q(s) quantifies how good a token-limited shortcut is for that position. Example. If we keep only (A2, B2) and zero all other digits, Q(((A2, B2), C1)) asks: "How often does the units digit computed from those two digits alone match the true units digit?" a concrete, measurable notion of shortcut strength.
The token-budget tree (Fig. 3). All subgroups that use exactly tokens (|A|+|B] = k) form level of token-budget tree. Parents at level k+1 strictly contain a child at level k (Token Inclusion). Under our masking protocol, adding observed digits cannot increase ambiguity for CjN giving the monotonicity
(8parent: 2 (Sechila)
whenever Schild C Sparent in tokens.
(4)
The root (all digits kept) attains Q= 1, while very small-token leaves approach chance quality (* 0.1 on uniform 10-way digits). How to read Fig. 3 Left-to-right within a layer, nodes differ in which digits they observe; top-to-bottom, layers differ in how many digits they observe, so moving upward generally (weakly) increases by equation
7

---

Published in Transactions on Machine Learning Research (01/2026)
A₁A2 x B, 2
A₁42 11
A₁42 x B2
A B2 Bz
A2 B₃ B2
A B
A2 x B1
A1 Bz
A1 B2
Figure 3: Tree Structure for 2-digits multiplication. Given an output position C, subgroups can be organized into a hicrarchical tree structure. Each layer represents the number of tokens used by the corresponding subgroup. We do not draw lowest layer (e.g.. A1) in this figure its quality equals 0.
Why edges are easy with few tokens. The U-shape comes from the existence (or lack) of high-quality, low-token subgroups across positions:
Least significant digit (C1). For A1 B₁ B2, the units digit depends only (A2, B2) (mod 10). Thus = ((A2, B2), Ci) (two tokens) achieves Q 1 even after masking higher digits, because those contribute multiples of 10.
Most significant digit (C4). With fixed-width imputs, order-of-maguituuee constraints and limited carry patterns mean that leading digits (e.g., A1, B1) already give a high-quality predictor for C₃ with small k.
Middle digits (C2,Cz) These integrate many cross-terms and carries. Any low-k subgroup necessarily omits essential contributors, so Q(s) is low until more tokens are included.
Ilustration. 27 x 38, the units digit is 7 x = 56 = 6 unchanged by masking the tens digits. For the x 102, so the most significant digit tightly constrained even
leading digit, x 47 (9 x 4) x 102 if not always exactly determined. By contrast, for 47 56 2632, the second digit (C2 3) depends on multiple cross-terms and carry; sceing only (Ag. B1) (7,5) leaves C2 ambiguous.
A Aggregating by budget & matching Fig. Cj:
Define the best quality achievable at budget k for position
Qx(C,)
max ((((A,B),C,)
IA -
For small h Qc(Ci) and (C) (MSB) are high, while Qc(C,) is low for middle j; hence the vector (Qu(C))...,, ( is U-shaped. As k grows, monotonicity in equation lifts all positions, but the middle digits catch up last. Link to the plot. In Fig. 4 the "2-digit-low htwp//mmddl/ffle mnihh series visualize exactly this: higher token budgets ("high") raise the entire curve while preserving the edge advantage at small budgets; the 23/4/5-digit-low/ curves show the same carly U-shape for longer inputs when the budget is tight.
Subgroup Quality O(s)
1.0
0.8
digit low
0.6
2 digigt-midddi middle
2 digit high
0.4
3-digitlow
4-digit-Low
0.2
5-digit-low
2
8
Position Index
Figure Position-level Subgroup Quality. The low-to-high order reflects the hierarchy in the tree structure. Similar trends in -5 digits with 2-digits (see Appendix B
8

---

Published in Transactions on Machine Learning Research (01/2026)
Phase 1
Phase 2
Phase n
Figure 5: Searching Program inside LMs. Different phases refer to different. stage of LMs fitting.
Next-token training tends to adopt the fewest-token, highest-Q shortcuts first. Phase in Fig. 5 highlights dark leaves: the model quickly fits C1 (units) and often Cm (leading) using low-k subgroups. As residual errors persist in middle positions, gradients drive the model upward to larger-token subgroups (Phase 2), and eventually toward near-root programs (Phase n). This climb up the eaken-budget tree explains both the early U-shape and its later flattening with more training. Operational You should see Fig- flatten over training checkpoints: edge positions plateau early, middle positions rise later as the model "unlocks" higher layers of token budget tree.
Learning dynamics as tree search (Fig.
Concrete 2-digit example. For A1A2 B1:2:
Oj: ((A2, B2), Ci) with k = yields Q = = easy early gains.
O2: any subgroup (e.g., ((A2, Bi), misses carry/cooss-ermss + low Q, needs higher k.
3. O4: ((A1, Bi), C4) already constrains magnitude + relatively high Q at small k.
Numbers. 48 39 1872, Ci=2 is fixed by (8, () alone, but C2-7 depends on 8x3, 4x9, and carry from 8x9, so observing only one of these pairs cannot nail C2.
Predictions and diagnostics. (i) If we cap usable tokens (e.g., stronger masking, context truncation), edge digits degrade least. (ii) As the cap relaxes (bigger context or reveal-ourriculum). the U-shape flattens from the middle outward. (iii) Data that amplifics mid-digit carries (hard negatives) accelerates the climb for middle positions. How to use this. To test the theory, gradually increase which digits are unmasked during training: you should see C1/Cm saturate early and C2/O's improve only when their requisite digits are revealed, matching the phase progression in Fig. 5
Takeaway. (ss) is a computable lens on shortcut viability under token constraints. Figures and together show the mechanism: models first exploit low-k / high-Q edge subgroups, then progressively integrale more digits to master the middle yielding the observed U-shaped accuracy curve and its evolution over training. Plain longuage. Edges are easy because there exist "cheap but good" rules for them; the middle is hard because it needs "expensive" rules that combine more digits, the model learns those later.
4 From Subgroup Quality to Subgroup Entropy
Although subgroup quality Q(s) is a useful measure for revealing a language model's tendency to use minimal tokens, it has limitations when compared to Kolmogorov complexity in UTM. Specifically:
The O(2" x n) complexity of calculating ((s) makes it computationally costly for long sequences.
It is difficult to compare arithmetic across tasks, and the connection between subgroup quality and error estimation is not clear
Therefore, we propose subgroup entropy as a complementary measure. It computationally efficient, and we plan investigate its error estimation capabilities across tasks.
In ST (Solomonoff 1964a), Solomonoff shows that the MSF (x: is upper-bounded by K(n)In2 upon convergence.

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Label space statistics with different rule perturbations. (()) represents the entropy of the label space, and [4] is the size of the label space. {C, represents all positions in output digits.
Ci
C2
Cs
Ca
Cs.
{O:
Task
Format
H(C)
[[[))
H(C)
H(C)
H(C)
[4]
H(C)
f(a,b) b) = a
A1Aa + B, C1CaCa
0.9710
3.3215
3.3219
179
7.2130
f(a.b) = a a+b+1
A1 + B, C1CaCa
0.9649
3.3215
3.3219
179
7.2130
f(a.b) 15
A1A2 + B, C1CaCa
0.9280
3.3214
3.3219
179
7.2130
f(a,b) +a+b+1l 115
A1 + B,
C1C₂Ch
0,9280
3.3214
3.3219
179
7.2130
f(a.b) b) = (a + mod 100
A1A2 + B. = C₁C₂
3.3214
3.3219
100
6.6432
f(a,b) = (a + b) mod 50
A1 + Bı B2 C1C²
2.3217
3.3219
50
5.6436
f(a,b) (a + b) mod 10
A1A2 B, C1
3.3219
10
3.3219
f(a,b) = x
A1 x B1 B2 C1C2OsC4
2.8979
3.3215
3.3160
3.0340
2621
11.1172
f(a,b)
6
A1 x Bi B3 C1CaC&CaOss
0.6873
3.2173
3.3215
3.2964
2.2227
2621
11.1172
f(a,b) b
A1A2 B, C,C.C,C,Cs
1.6030
3.3020
3.3204
3.2234
2.2227
2621
11.1172
f(a. b) x
ArA2 x B, C,C,C,C,Cs
2.5811
3.3202
3.3151
3.2235
2.2227
2621
11.1172
f(a, b) = (a x b) mod 100
A,Aa x B2 CiC₂
3.3160
3,0340
100
6.2912
f(a, b) (a x b) mod 50
A: x B, B2 C₁C₂
2.3210
3.0340
50
5.3494
f(a. b) (a mod 10
Ai B, B2 C₁
3.0340
10
3.0340
4.1 Subgroup Entropy as "Cheap" Error Estimation
Scttings. We first deliberately perturb the underlying data generation function f to observe the correlation between subgroup entropy H' and accuracy. We consider addition f(a,b) a + b and multiplication f(a, b) = x b as our baselines. For addition, the perturbation is defined as f(a. b) = a - b + Ac, where Ac E f1, 15, 115) corresponds to offsets at different cositions/magnitudeen For multiplication, the perturbation is defined as f(a, b) = x bx with {2, 4, 8} for analogous reasons. Additionally, we incorporate modular addition and multiplication as further perturbations. Table [ summarizes the label-space statistios after applying perturbations. We then fine-tune Gemma-2-22 2-2B (Team et al., 2024) and Llama-3.1-8B (Dubey et al., 2024 on data gencrated by these perturbation functions to analyze how subgroup entropy H' Hs) evolves.
Entropy Accuracy Analysis. The results in Table [ show that both Gemma-2-20 2-2B and Llama- 3.1-8B produce consistent outcomes across two rule perturbation methods and three sctups. Interest- ingly, LLMs handle arithmetic like 13 x 10 = 520 similarly to 13 x 10 = 130 when the subgroup en- tropy (s) remains fixod. This demonstrates that tasks with similar Hs(s) share comparable error bounds. For modular addition and multiplication with varying modulus values, reducing the entropy size improves performance in both cases. These findings suggest that arithmetic lasks with lower (s) in the subgroup space are easier to learn and result fewer errors. This phenomenon con- firms that subgroup entropy better measure
Table 2: Test Accuracy difference A on perturbed opera-
tions. Data Generation Function
[f({})
Gomma-2-211
Alama-3.1-813
(a,b)) =ab
7.21
f(a, a 6++1
7.21
-0.1%
-0.1%
f(a, a
7.21
-0.9%
40.1%
fla
115
7.21
-1.4%
+0.7%
f(a, 6) (a 6) mod 100
6.64
+10.1%
+3.7%
f(a, 5) (a mod 50
5.61
+13.1%
+6.7%
f(a b) (a mod
3.32
+26.1%
+13.7%
f(a, 6) xxxx
11.12
[(a, b)
x
11.12
-111%
-2.7%
[(a,
-1.7%
+0.7%
[(a, 6)
11.12
+0.2%
-3.7%
[(,, b) x 6) mod 100
6.29
+7.1%
+3.88
M(, b) = (a x b) mood 50
5.35
+12.1%
+5.3%
for estimating errors in arithmetic learning tasks.
fa. 6) (a x 6) mod 10
3.03
+18.9%
+10.7%
What Table 1 reveals (entropy shifts under perturbations). Three consistent patterns emerge. (i) Additive translations preserve total label entropy while redistributing it across positions. For f(a a + b and b Ac, the total entropy remains H(C)= = =721130, but the lowest digit becomes slightly more predictable as Ac grows (e.g., Ci drops from 0.9710 to 0.9280), whereas C2/O3 stay near 3.32; this reflects a carry pattern shift localized to early positions. (ii)
2 (s) represents H(s) summed all output pasitions C.
10

---

Published in Transactions on Machine Learning Research (01/2026)
Multiplicative scalings largely conserve total entropy while shifting where entropy lives. For b vs. a b x X, the total stays 11.)21 but entropy mass migrates toward higher positions (a new Cs appears with 2.22 bits) and the units digit entropy rises monotonically with À (C1: 0.69 - -.0--2.5 for À = 2,4, 8), capturing how scaling changes residue classes and carry dispersion across digits. (iii) Modulor reductions compress the label space and thus the total entropy roughly to logg [c]. For addition, (mod 100, 50, 10) yields totals 6.64, 5.64,3.39 (very close to log2 100, log2 50, log, 10), consistent with nearly uniform residues; for multiplication the totals are slightly smaller (6.29, 5.35,3.03) due to non-uniform multiplicative residue distributions (e.g., ligi- biases), explaining why modular tasks empirically become easier as [C] shrinks.
4.2 Extend Study CoT: A Symbolic View
Building on subgroup entropy's offectiveness in error ostimation for arithmotic tasks, we extend our experiments from simple arithmetic to a chain-of-thought (CoT) setting (Wei et al. 2023). CoT involves multi-step reasoning to solve a single problem. For example, in arithmetic learning, solving multiplication problem using CoT means breaking it down into smaller steps arrive at the correct answer. Recent studies have shown that RL-based CoT improves reasoning abilitics in advanced LMs such OpenAI ol (OpenAl 2024 and DeepSeek-rl (DeepSeek-A1 et al. 2025
Settings. We still take multiplication as the baseline. While multiplication mathematically well-defined, there are multiple methods to perform the calculation. Historically, four methods stand out as the most representative: Standard Multiplication, Repetitive Addition, the Lattice Method, and Egyptian Multiplication (details in Appendix C.1 These methods correspond to four distinct CoT paths. We finc-tune Gemma-2-2B and Llama-3-8B on data from these CoT paths to observe how H'()) evolves.
Table 3: Performance comparison of different CoT paths. Different CoT split the original tasks into different CoT steps.
CoT Type
# CoT Step
Avg. (s)
Avg. Q(s)
Accuracy (%)
Original
11.1172
9.87
73.12
Standard Multiplication
3
7,8431
7.12
86.32
Repetitive Addition
5.8313
6.03
91.31
Lattice Method
5.1130
5.41
94.41
Egyptian Multiplication
3.2130
4.02
98.31
The results of the performance comparison between four different CoT methods are shown in Table 3 Using no intermediate steps (Original), the task yields an average entropy of 11.12 and a test set accuracy of 73.12%. This serves as a baseline to compare the effectiveness of CoT methods. Two key observations can be drawn:
Results.
Splitting into More Steps Improves Performance: Decomposing a task into multiple steps, as demonstrated by the Egyptian Multiplication method, leads improved performance compared the baseline. This likely occurs because splitting the task results in lower average entropy H'(s) per sub-task.
Strategic CoT Splitting is Crucial: Comparing standard multiplication to repetitive addition reveals that even when a task is split into a fixed number of steps, the choice of splitting strategy significantly impacts performance. Choosing a split that yiclds lower average entropy H'(s) per sub-task results in superior overall performance.
The subgroup quality (s) proves to be effective at estimating and correlating CoT errors. By focusing on the entropy of individual steps in the reasoning process, this approach offers a more detailed and effective way to evaluate and optimize performance in complex reasoning tasks. These findings demonstrate the potential of the subgroup induction framework to advance reasoning-intensive math tasks.
11

---

Published in Transactions on Machine Learning Research (01/2026)
4.3 Discussions
Overall, propose two measures to assess subgroup complexity: subgroup quality Q(s) and subgroup entropy (s). These measures complement each other. Subgroup quality is best suited for understanding learning mechanisms, revealing how LMs initially try to use fewest tokens and gradually incorporate more tokens if needed. However, it is computationally expensive and less effective for error estimation. On the other hand, subgroup entropy is less effective at capturing learning dynamics but excels in error estimation across tasks, including both simple arithmetic and CoT settings. Together, these two measures provide a robust framework for evaluating subgroup complexity.
5 Related Work
Understanding Arithmotic Learning in Transformer Research on understanding arithmetic primarily in previous work is to identily causal correlations wetween model components and outpuls. [toolo et al. (2023 identify key attention layers responsible for arithmetic learning using causal mediation analysis (CMA), a weight perturbation method that observes changes in output. Similarly, Hanna et al. (2023) and Wu et al. (2024) explore causal abstraction concepts at different model scales, specifically 0.1B and 7B parameters, respectively. More recently, ZZhang et al. (2024) isolate attention heads and fine-tune them for improved performance at lower cost. While these studies have made progress in understanding how LMs perform arithmetic at component level, our rescarch provide another perspective from algorithm learning theory.
Large Language Models and Solomonoff Induction Solomonoff induction is widely regarded as one of the most powerful predictors in the field of universal prediction, providing a theoretical foundation for optimal Bayesian inference in algorithmic probability (Solomonoff 1964a b). Meanwhile, large language models have
sparked discussions about whether they approximate universal Turing machines (Chen et al. 2018 Lu & Lu 2020 Stogin el al. 2022 Mali et al.] 2023) and whether their learning behavior resembles Bayesian updating Ortega et al., 2019 Arora et al. 2024).. And some previous work also discuss the relationship between transformer's bohavior and Solomonoff induction (Young & Withrock 2024 Grau-Moya et al. 2024).
These debates have inspired us to develop a practical framework for understanding the mechanisims underlying language models.
Arithmetic Reasoning in LLMs
Mathematica ical reasoning has been a longstanding area. of research in
natural language processing (NLP) (Kushman et al, 2014 Huang et al. 2016 Wang et al., 2017 Thawani
et al.] 2021: Sundaram et al.) 2022 Guo el al.]
2024). However, LLMs still face challenges with basic
calculations and remain vulnerable to adversarial examples or perturbations, where minor changes in problems can result in incorrect answers (Zhou et al. 2023 Xie et al.) 2024h. Several previous efforts have aimed to improve arithmetic learning in LLMss. Lee et al. (2023) trained a 10.6M NanoGPT (Karpathy) 2022) model to learn arithmetic by carefully curating the data format, explicitly expanding each step using a method termed Scratchpad, which achieved remarkable performance compared to GPT-2 XL (Radford et al. 2019 Yang et al. (2023 finc-tuned MathGLM with a sufficient training dataset, demonstrating its capability to solve 5-digit multiplication. Deng et al. (2023 (202)) further advanced this field by internalizing the CoT process, hiding detailed steps in a scheduled manner, enabling GPT-2 small solve 9-digit multiplication after multiple training runs.
6 Conclusions
In conclusion, our work introduces the novel framework of subgroup induction, offering a pragmatic adaptation of Solomonoff induction tailored to arithmetic learning in language models. By leveraging subgroup quality and entropy as complementary measures, we uncover a U-shaped learning pattern that reflects Occam's 's Razor principles in LMs. This innovative approach bridges theoretical universal prediction with modern neural architcctures, providing robust insights into arithmetic learning mechanisms. Our findings emphasize the practicality of our framework in addressing complexity and advancing reasoning research, establishing a foundational step forward in understanding and improving the capabilities of LMs in arithmetic tasks.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Acknowledgements
We thank the anonymous reviewers for their thoughtful feedback and constructive suggestions, which helped improve this work. This project is supported in part by the Rice Ken Kennedy Institute Research Award # #0881025. We also thank members of the CHILI Lab for helpful discussions and support throughout the project.
References
Anthropic. Claude, 2024. URL httpss//ww..attirppp...omm
Aryaman Arora, Dan Juralsky, Christopher Potts, and Noah D. Goodman. Bayesian scaling laws for in-context. learning, 2024. URL httpps///arai..oo///////2......
Yining Chen, Sorcha Gilroy, Andreas Malctti, Jonathan May, and Kevin Knight. Recurrent neural networks weighted language recognizers. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 2261-2271, New Orlcans, Louisiana, June 2018. Association for Computational Linguisties. doi: 10.18653/v1. / URL https://aclanthologg org/N18-1205/
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021. URL Inttps///axxi> hrr.i..vviioo////////////o 2bss/110..1... 14168
DeepSeek-AT, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, and et al. Decpseck-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL
https: httpa////xxwv..oraabbaaaa.
12948
Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. Implicit chain of thought reasoning via knowledge distillation, 2023. URL https://arx/v.org/ wttp://ww.rr//b///33333 absgg/abs22111 01460
Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to internalize cot step by step, 2024. URL httpps////axvvrrr//a///2/3. wrw..bs/24/5..5..... 14838
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloc Bi, and et al. The llama 3 herd of models, 2024. URL https://arxie http:://arxiv.orr//aa///22222111 21783
Jordi Grau-Moya, Tim Genewein, Marcus Hutter, Laurent Orseau, Grégoire Delétang, Elliot Catt, Anian Ruoss, Li Kevin Wenliang, Christopher Mattern, Matthew Aitchison, and Joel Veness. Learning universal predictors, 2024. URL http:://arxxvoor///ab///////// wwwp://////.......//////
14953
Siyuan Guo, Aniket Didolkar, Nan Rosemary Ke, Anirudh Goyal, Ferenc Huszár, and Bernhard Schölkopf. Learning beyond pattern matching? assaying mathematical understanding 1lss, 2024. URI https:
Tlarsiv.org/abs//4000 15485
Michael Hanna, Ollic Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?? Interpreting mathematical abilities in a pre-trained language model, 2023. URL
13

---

Published in Transactions on Machine Learning Research (01/2026)
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL htttps://arxe org/abs/
F166.09685
Danqing Huang, Shuming Shi, Yewn Lin, Jian Yin, and Wei-Ying Ma. How well do computers solve math word problems? large-scale dataset construction and evaluation. In Katrin Erk and Noah A. Smith (ods.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 887-896, Berlin, Germany, August 2016. Association for Computational Linguistics. doi:
N0.1863//11P11--1 1084. URL. jttpp:://aclanhhooooy.oorg
Andrej Karpathy, Andrej karpathy's lightweight implementation of medium-sized gpts. https: //ghthb com/karpathy/nanodPT 2022. Accessod: 2024-09-28.
Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. Learning to automatically solve algebra word problems. In Kristina Toutanova and Hua Wu (eds.), Proceedings of the 52nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 271-281, Baltimore, Maryland, June 2014. Association for Computational Linguisties. doi: 10.3115/v1/P14-U0 URL https://acllanthol..0 111010/100000//1--11100
Nayoung Lee, Kartik Sreenivasan, Jason D. Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers, 2023. URL https://arxiv.orr//aat obwp:///rgga//b/////////o http///bs/23077 2ttp///307703381111
Yulong Lu and Jianfeng Lu. A universal approximation theorem of deep neural networks for expressing
probability distributions, 2020. URL httpp:://aarxiv.orggab////////// 08867
Ankur Mali, Alexander Ororbia, Daniel Kifer, and Lee Giles. On the computational complexity and formal hierarchy of second order recurrent neural networks, 2023. URL httpps://rxxv.orr//// www.ooggg/bb//////// 14691
OpenAI. Learning to reason with llms, 2024.
URL https://openaa. catpmnai..mmmiieeeeeee /ndex/
Learning-to- tore--ttorrr--iitttteeee wimms
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel BernadeU-Shapiro,, Christopher Berner, Lenny BogdonofT, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, and et al. Gpt-4 technical report, 2024. URL attps://arxiva http://wxxvvv.o//aaa//
Pedro A. Ortega, Jane X. Wang, Mark Rowland, Tim Genewein, Zeb Kurth-Nelson, Razvan Pascanu, Nicolas Heess, Joel Veness, Alex Pritzel, Pablo Sprechmann, Siddhant M. Jayakumar, Tom McGrath, Kevin Miller, Mohammad Azar, Ian Osband, Neil Rabinowitz, András György, Silvia Chiappa, Simon Osindero, Yee whye Teh, Hado van Hasselt, Nando de Freitas, Matthew Botvinick, and Shane Legg. Meta-learning of sequential strategies, 2019. URL https: http://aaxxi...r/bbbbaaa/ wttpg//g///g////eee/e/e//// 03030
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Tyy Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Ray J Solomonoff. A formal theory of inductive inference. part i. Information and control, 7(1):1 -2, 1964a.
Ray J Solomonoff. A formal theory of inductive inference. part ii Information and control, 7(2):224 254, 1964b.
John Stogin, Ankur Mali, and C Lee Giles. A provably stable neural network turing machine, 2022. URL https://arxiv www.arwvw....aaaaaaaa www//rag///aaeeeeeeeoooeee 03651
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In Houda Bouamor, Juan Pino, and Kalika
Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7035-7052, Singapore, December 2023. Association for Computational Linguisties. doi: 10.18653/v1 v1/2023. emnlp-main Rttllhttts://aclll htttpss://claaatholoogyy wwt.p/iy..rg/222.e..
http//02..emmlpmmaaiee emnlp-
14

---

Published in Transactions on Machine Learning Research (01/2026)
Sowmya S. Sundaram, Sairam Gurajada, Marco Fisichella, Deepak P. and Savitha Sam Abraham. Why are nlp models fumbling at elementary math? survey of deep learning based word problem solvers. ArXic, abs/2205.15683 2022. URL http::///pp..smmaiccscs... www.aaapu.rrC/CCI.III.... : 249209981
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabcla Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, and el al. Gemma 2: Improving open language models al. a practical size, 2024. URL https: attps://arxiv .rgg abaaaaaaaaaa 2408.00000
Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely. Representing numbers in NLP: a survey and a vision. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkami- Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Humon Language Technologies, pp. 644-656, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/va 000653/vl/25 hatp///2maacmmmmiinnn..mm maacl-main.55 53. URL
org/2021 www.laaalaaaaa -miin.5
Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 845-854, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10.18653/v1/DD/. 1088. URL
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL wwwpp/wwww..r////////000000
Zhengxuan Wu, Atticus Geiger, Thomas Ieard, Christopher Potts, and Noah D. Goodman. Interpretability http://aai/v.rrr/a//////////0000
scale: Identifying causal mechanisms in alpaca, 2024. URL
Roy Xie, Chengxuan Huang, Junlin Wang, and Bhuwan Dhingra. Adversarial math word problem generation, 2024. URL https://arxiv. www.w/orr///bba httgg/ab///a 0aabs/2242.110 17916
Zhen Yang, Ming Ding, Qingsong Lv, Zhihuan Jiang, Zehai He, Yuyi Guo, Jinfeng Bai, and Jie Tang. Gpt. can solve mathematical problems without a calculator, 2023. URL atttp:://aravv.or//bbb//// 03241
Nathan Young and Michael Witbrock. Transformers as approximations of solomonoff induction, 2024. URL
fttps://arxiv.org/abbe www./r/g////////oo/ 12065
Wei Zhang, Chaoqun Wan, Yonggang Zhang, Yiu ming Cheung, Xinmei Tian, Xu Shen, and Jieping
Ye. Interpreting and improving large language models in arithmetic calculation, 2024. URL https
/arxiv.org/abs/24400
Daniel Zhang-Li: Li, Nianyi Lin, Jifan Yu, Zheyuan Zhang, Zijum Yao, Xiaokang Zhang, Lei Hou, Jing Zhang, and Juanzi Li. Reverse that number! decoding order matters in arithmetic learning, 2024. URL
http//axxvv.or/aabb/
Zihao Zhou, Qiufeng Wang, Mingyu Jin, Jie Yao, Jianan Ye, Wei Liu, Wei Wang, Xiaowei Huang, and Kaizhu Huang. Mathatlack: Atlacking large language models towards math solving ability, 2023. URL
https://arxiv. http:///axxvv..og/bbbaeas htt////bbb///..../ 01688
A Experiment Setup
In this section, we detail the experiment setup used in the main body of our work. Our experiments are designed to investigate the capabilities of language models in learning and performing arithmetic tasks, with a focus on understanding the underlying mechanisms of arithmetic reasoning. By carefully selecting the domain, models, training methodologics, and datasets, we aim to provide a comprchensive analysis of how language models can be adapted to handle arithmetic operations effectively. Below, we describe each component of our setup in detail.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Domain. We select addition and multiplication the fundamental operations for experiments, following previous work (Lee et al. 2023: Deng et. al. 2023 2024). We do not consider division or subtraction, subtraction can be viewed as a similar operation to addition, and division introduces lloting-pooin numbers, which involve more complex formatting issues in this task. Therefore, we primarily focus on multiplication and aim to understand it in depth. However, by definition, our subgroup induction framework suitable for all arithmetic tasks.
Model. To investigate arithmetic learning at scale, we selected two open-source LLMs, Gemma-2222 (Team ct al. 2024) and Llama-3.1-8B (Dubey et al., 2024). Both models are top performers in their respective calegories and excel at language tasks. We did not choose GPT-4o (OpenAl et al.) 2024) or other proprietary LLMss due to concerns that they may internally integrate function calling (e.g., invoking APls executing Python programs), which could affect the experimental setup.
Training Details. We use LoRA Hu et al. 2021 | to optimize memory usage during the training of language models. The LoRA rank is set to 8, and we employ rank-stabilized LoRA in our setup. We set the learning rate to 3e-4, the number of epochs to 12, and the batch size to 16. Additionally, we use gradient checkpointing with a step size of 4 to save memory.
Data. For n-digit arithmetic, we consider the optimal dataset size. Specifically, for 2-digit multiplication, we have 8,100 data points in total, and for 3-digit multiplication, we have 810,000. We split the dataset into train, development, and test sets with a ratio of 8:1:1. We train the language models on the training set, select the best checkpoints based on validation loss, and use them to evaluate the test set accuracy.
Conventional Data Format. We directly train the model to predict the output (e.g., 130) given the input operands and the operator (e.g., x 10). e add one space between each digit to ensure tokens are split. into individual digits We do not use chain-of-thought (CoT) (Wei al. 2023) or other prompting strategies to enforce the model focus on arithmetic learning in main experiment body. But we will discuss how subgroup entropy could be suitable metrics for quantifying CoT path.
B Subgroup Quality for High-digits Multiplication
B.1 Criteria for Low/Middle/High Node Selection
We define the low node as subgroups with two token positions, such as A₁ which are located closest to the leaf nodes in the tree structure. The middle node selected as subgroups with [m] token positions digit arithmetic. Subgroups with closer to the rool node. In summary, the middle node resides at the intermediate level of the tree structure, the low node is near the leaf level, and the high node is near the root level.
token positions are classified high nodes, which are positioned
B.2 Subgroup Quality for 3/4/5-digits Multiplication
As shown Figure G we can also observe a similar trend in these multiplications with the language model (LM) learning arithmetic, characterized by U-shaped learning. In high-digit multiplications, the subgroup quality for the middle digits remains very low for lower nodes, while higher nodes exhibit better quality. This phenomenon strongly reveals the underlying mechanism of language models' learning dynamics: they tend to use more and more tokens in predictions to reduce the loss during gradient descent. We believe this is an imporlant finding that may be helpful for future research.
C Are Large Language Models Implicit Calculators?
In this scction, we explore whether LLMs utilize partial products enhance their arithmetic calculation capabilities, particularly in the context of multiplication. is important note that while multiplication is well-defined mathematically, the process of multiplication calculation is not limited to traditional methods
16

---

Published in Transactions on Machine Learning Research (01/2026)
Singang Q40
Sipporg Qualey
Foaiioo Indos
Prsiian Index
Podiio Ioles
(a) 3-digits Multiplication
(b) 4-digits Multiplication
(c) 5-digits Multiplication
Figure 6: Subgroup quality for 3/4/ 4/i/is multiplication given different node.
defined textbook. Thus, examining only one calculation method presents flawed experimental design that is vulnerable to exploitation. We selected four calculation methods that are representative to cover the major approaches to multiplication.
C.1 Historical and Modern Multiplication
In terms of multiplication, four different calculation methods are most representative from history to now: Standard Multiplication, Repetitive Addition, Lattice Method, and Egyptian Multiplication.
M1: Standard Multiplication
In standard multiplication, we multiply each digit of one number by each digit of the other number, and then sum the results appropriately:
12 x 34 = 12 x (30 + 4) 12 x 30 + 12 4
360 + 48 408
M2: Repetitive Addition thirty-four times:
Multiplication can be interpreted as repeated addition. For 12 x 34, we add 12
12 x 34 12 + 12 + 12 + 12 (34 times)
408
M3: Lattice Method In the lattice method (or grid method), we place the numbers along the top and side of grid, perform single-digit multiplications, and then sum along the diagonals:
12 34 10 30 300
10 x 4 40
x 30 60
2 x 4 = 8
Summing the results: 300 + 40 + 60 + 8 = 408
M4: Egyptian Multiplication Egyptian multiplication computes the product by doubling the multiplicand and adding the results corresponding to the powers of two that sum to the multiplier. For 12 x 34:
12 x 34 12 x 12
12 x 24
12 x = 48
12 96
12 16 192
12 x 32 384
Summing the selected results: 24 + 384 408
Since 34 2 2232 32, select the results for 12 16 and 12 8, and summing these gives the final product.
17

---

Published in Transactions on Machine Learning Research (01/2026)
Table 4: Inductive and deductive accuracy difference A.
Gemma-2-28
Llama-3.1 3-B-
Standard
Lattice
Repetitive
Egyptian
Standard
Lattice
Repetitive
Egyptian
Task - Partial P.
+4.1%
+6.8%
-9.00%
+3.6%
+40.6%
+40.8
-99.0%
+29.6%
Partial Task
6.11%
-1.77%
-20.3%
-9.6%
-3.7%
-0.2%
-0.9%
-2.7%
C.2 Examining Partial Product in Arithmetic Learning
To investigate whether LLMs generate partial products during arithmetic learning, we employ a set of diagnostic tasks as an approach to trace. We fine-tune Gemma-2-2B and Llama-3.1-8B on two-digit multiplication, observing changes in accuracy on diagnostic sets before and after fine-tuning (Task Partial Products). Subsequently, we fine-tune the LLMs on these diagnostic sets and examine how their accuracy on the multiplication task changes (Partial Products + Task).
Table Diagnostic sets with four calculation methods.
Method
Diagnostic Scts
Standard Multiplication
Paud {Aı By B2, A2 B2, Bi A1 B2 x A1
Repetitive Addition
Pra { B,B2 A A1 Em B1 B2B2)
Lattice Method
Plattice = {10 x B10, A10 x B2, A₂ x B10, A2 X B2}
Egyptian Multiplication
P Pegyptian 12h A142 E 0,1...., [logz (2)
We probe language models' partial product four different directions. As shown Table 5 for a task formatting like A142 B, B2 = C,C2C3C4, we would generate diagnostic test for each algorithm.
Accuracy Identifying Partial Products According to the results in Figure we found that standard multiplication, the lattice method, and the Egyptian method significantly improved in identifying partial products after fine-tuning, with gains of +17.45%, +18.35%, and +10.45% respectively. However, for repetitive addition tasks, LLMs failed to identify partial products, achieving an accuracy of only about 5% after fine-tuning.
A Deeper Look into Calculations Do the results showing increased accuracy across three paths really imply that partial products are used in arithmetic learning? We have two arguments against this interpretation. First, if LLMs genuinely use partial products to learn arithmetic, it is likely that they only use one calculation path at time. Thus, the simultaneous improvement across three paths (standard, lattice, and Egyptian) is unusual. Second, if LLMs employ specific path to compute partial products, this process should be demonstrated as bidirectional. Specifically, LLMs fine-tuned on a task should be able to identify partial products (inductive), and conversely, mastering partial products should enhance lask learning (deductive). However, we currently have results for only one direction, lacking evidence for the other. Therefore, we extend our experiments another dircction.
Accuracy on Identifying Tasks We fine-tune two LLMs on diagnostic sets and present the results of identifying tasks before and after fino-tuning in Table 4 Our findings reveal that, finc-tuning specifically partial products does not enhance task learning. Instead, it results in performance drop across all four calculation paths for both models. This indicates that pre-learning partial products does not aid in arithmetic learning. The improved ability to recognize partial products appears to stem from the symbol learning process (note that the standard partial product A₁ x B, is a sub-portion Aj x B, B2, similar lattice and Fgyptiaa methods) rather than being an intrinsic computational method used by the models.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Partial Products Identication Accuracy
1.0
0.975
0.939
0.791
0.8
0.737
0.591
Values
0.6
0.562
0.4
0.298
0.201
0.2
0.096
0.045
0.0
Std
Rep add.
Lattice
Egyptian
Acc
Before FT
After FT
Figure 7: Partial products identification accuracy before and after fine-tuning on tasks. Scores are reported
average of Gemma-2-22 2-2B and Llama-3.1-8
19


Reviews:
[{"review_id": "agKi2RzK4r", "reviewer": "Reviewer_DgXP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates how Language Models (LMs) handle numerical reasoning using addition and multiplication arithmetic tasks. The core approach involves defining \"subgroups\" - specific subsets of input variables - to analyze the LM's performance. First, the authors measure the accuracy (\"subgroup quality\") of the LM for different subgroups within these tasks. Second, they introduce an entropy-based metric (\"subgroup entropy\") which they show to be an efficient predictor of this \"subgroup quality.\"\n\nThe authors make the key finding that contrary to some prior expectations, the LM exhibits a distinct U-shaped trend regarding the prediction of the arithmetic result. This means the model learns the least significant digits (last digits) and the most significant digits (first digits) more easily. However, Intermediate digits appear harder to predict, suggesting their interactions are more complex for the LM to learn. The authors interpret this behavior as the model exploiting \"shortcuts.\" For example, predicting the final digit only requires looking at the last two input digits, while the first digit often has limited variability for fixed-size inputs, making the middle digits the more challenging.\n\nFinally, the paper demonstrates a correlation between the \"subgroup quality\" (accuracy) and the proposed \"subgroup entropy\" measure. The utility of the entropy measure is further illustrated through experiments mimicking chain-of-thought (CoT) reasoning.\n\n**Strengths:**\n\nI found this paper engaging and well-executed. It provides a very clear demonstration of the simplicity bias in LMs, particularly when applied to these arithmetic tasks. The paper is logically structured and easy to follow. I find the method of subgrouping arithmetic tasks to be neat and effective for highlighting how LMs tend to pay attention to specific (easy) statistical patterns rather than attaining a general arithmetic rule.\n\n**Minor Comments and Open Questions:**\n\nI would appreciate a bit more detail on the exact methodology for the chain-of-thought (CoT) experiments. A brief explanation in the appendix would be helpful, especially for readers who might not be deeply familiar with the nuances of CoT. This would further improve the self-sufficiency of the work.\n\nThe authors focused on multiplication and addition (noting that subtraction is similar to addition, and division introduces complexity with floating points). I wonder if the authors considered exploring lower bases, such as binary arithmetic? I suspect that the observed effects (like the U-shape bias) might become even more pronounced in binary, potentially allowing for a clearer connection between the model's behavior and theoretical concepts like code-length or minimum description length.\n\nWhile the manuscript is generally very well written, the abstract seems slightly less polished than the body of the paper. Specifically, the phrase “[…] possible shortcuts to fit the stats of dataset to solve arithmetic tasks” could be tightened up. Also, there is a typo in the sentence: “In this paper, we argues […]”. These are minor points, and I trust the authors to decide whether to make adjustments.\n\n**Disclaimer:** I am no expert in the field of language-models and potentially missed important nuances in my review.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The provided experiments are plenty and clearly support the claims made.", "audience_yn": "Yes", "audience_explanation": "The paper offers tangible, interesting, insights into the learning behaviors/trends in LMs, which I assume to be valuable to the ongoing discussion regarding arithmetic capabilities of LMs.", "requested_changes": "None.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fqeq9lUWuh", "reviewer": "Reviewer_4ehb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### Summary\n\nThis paper argues that Large Language Models (LMs) performing arithmetic tasks do not learn the underlying procedural algorithms but instead act as **\"greedy symbolic learners\"**. To demonstrate this, the authors introduce the **subgroup induction** framework, inspired by Solomonoff Induction, which analyzes \"subgroups\"—minimal shortcut mappings from a small subset of input digits to a single output digit. The paper's primary contribution is the discovery of a distinct **U-shaped accuracy curve** in multi-digit multiplication: LMs quickly master the first and last output digits but consistently fail on the digits in the middle. The authors show this U-shape perfectly mirrors the viability of the simplest (fewest-token) shortcuts, as measured by their proposed **subgroup quality ($Q(s)$)** metric. This suggests a \"low-to-high\" token learning mechanism, where LMs first learn easy, low-token symbolic mappings before incorporating the more complex, multi-token patterns required for middle digits. The framework is extended using a second metric, **subgroup entropy ($H(s)$)**, which is shown to be an effective predictor for the performance of different Chain-of-Thought (CoT) reasoning paths.\n\n---\n\n### Strengths\n\nThe paper's primary strength is its **originality** and **clarity** in methodology. The \"subgroup induction\" framework is a novel and highly intuitive method for empirically formalizing and testing the \"shortcut learning\" hypothesis in LMs. Grounding this concept in the principles of Solomonoff Induction and Occam's Razor provides a strong conceptual foundation. The **quality** of the experimental work is high; the discovery of the U-shaped accuracy curve is a robust finding, compellingly validated across different model sizes (Gemma-2-2B and Llama-3.1-8B) and multiple task complexities (3- to 5-digit multiplication). The \"token-budget tree\" and the direct comparison of the U-shaped accuracy (Fig. 2) to the U-shaped subgroup quality (Fig. 4) provide a clear, \"smoking gun\" visualization of the \"fewest-tokens-first\" learning mechanism. The paper's **significance** is substantial, as it offers some of the most compelling evidence to date in the \"pattern matching vs. procedural reasoning\" debate, strongly suggesting LMs are not learning human-like algorithms but rather a hierarchy of symbolic shortcuts. Finally, the extension of the framework to evaluate Chain-of-Thought paths using subgroup entropy ($H(s)$) is a significant and practical contribution to reasoning research.\n\n---\n\n### Weaknesses\n\nThe paper's central claim about \"arithmetic\" is based almost exclusively on multiplication. This focus is understandable, but it limits the generalizability of the findings. It is unclear how the subgroup induction framework would apply to operations with fundamentally different properties, such as division (which introduces floating-point numbers) or subtraction (which involves a \"borrowing\" mechanism rather than \"carrying\"). The paper also frames its findings as \"symbolic shortcuts\" in opposition to \"algorithms\", but a more nuanced discussion would be beneficial; one could argue that the \"low-to-high\" token-budget-climbing is itself a type of greedy algorithm, just one that is qualitatively different from standard human procedures. Lastly, the specific operationalization of **subgroup quality ($Q(s)$)**, which relies on masking non-included digits to zero, is a critical design choice. The authors could strengthen their claim by discussing why this masking protocol is the most appropriate proxy for a \"shortcut\" and justifying why other forms of input perturbation (e.g., masking to a special token, or averaging) were not used.", "claims_and_evidence_yn": "Yes", "claims_explanation": "### Claim 1: LMs are \"greedy symbolic learners\" that follow a \"fewest-tokens-first\" heuristic.\n\nThe authors present a \"U-shaped accuracy curve\" (Figure 2). This shows that models (Gemma-2-2B and Llama-3.1-8B) are highly accurate at predicting the first and last digits of the product but fail significantly on the middle digits.\n\n### Claim 2: Subgroup entropy ($H(s)$) is an effective and \"cheap\" measure for estimating task difficulty.\n\nThe authors conduct experiments by perturbing the arithmetic rules (e.g., $a \\times b$ vs. $a \\times b \\times 8$) and testing model performance (Table 2).\n\n### Claim 3: The subgroup induction framework can be extended to analyze multi-step reasoning (CoT).\n\nThey compare four different CoT methods for multiplication (e.g., Standard, Lattice, Egyptian) and measure both the average subgroup entropy per step and the final accuracy (Table 3).", "audience_yn": "Yes", "audience_explanation": "###Addresses the \"Reasoning vs. Pattern Matching\" \n\nDebate: The paper's core question—whether LMs \"learn to truly compute, or do they simply master sophisticated pattern matching\" —is one of the most fundamental and contentious topics in the field. The TMLR audience is deeply invested in understanding the true capabilities and limitations of LMs, and this paper provides strong, empirical evidence for the \"pattern matching\" side of the argument.\n\n###Provides a Novel Analysis Framework: \n\nThe paper introduces \"subgroup induction\" as a practical, theory-inspired framework for analyzing LM behavior. Researchers are constantly seeking new tools and methodologies to interpret and understand how models learn. This framework, which bridges the theory of Solomonoff Induction with empirical testing, is a novel contribution in itself.", "requested_changes": "### Questions\n\n1.  The paper convincingly demonstrates the final U-shaped accuracy curve (Fig. 2) and proposes the \"low-to-high\" token learning dynamic (Fig. 5) to explain its formation. Have the authors analyzed model checkpoints *during* training to empirically verify this dynamic, as suggested by the \"Operational cue\"? Does the accuracy on edge positions (requiring low-token subgroups) saturate much earlier in the training process than the accuracy on middle positions (requiring high-token subgroups), as the \"Phase 1 $\\rightarrow$ Phase n\" diagram implies?\n2.  The paper notes that calculating subgroup quality $Q(s)$ is computationally costly. Could you clarify the practical methodology used to generate Figure 4? Given the $(2^{2n}-1)\\times2m$ total subgroups, was this analysis restricted to low-token-budget subgroups (small $k$), or was a specific sampling strategy employed to make this computation tractable?\n3.  The results showing that subgroup entropy $H(s)$ *correlates* with CoT path success are very promising. Do the authors see a path to using this *prescriptively* rather than just descriptively? For instance, could an LM be guided during inference (or through reinforcement learning) to dynamically generate a reasoning path that actively minimizes the average subgroup entropy $H'(s)$ of its intermediate steps?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Uk2HLYDVkJ", "reviewer": "Reviewer_a3oT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduced the novel subgroup induction framework, adapted from Solomonoff Induction (SI), to empirically test whether Language Models (LMs) perform arithmetic by genuine computation or by mastering symbolic patterns. The paper argues strongly for the latter, concluding that LMs act as greedy symbolic learners, prioritizing the simplest possible shortcuts (symbolic patterns) to fit the dataset statistics.\n\nThe core contributions demonstrating this pattern-matching/memorization behaviour include:\n\n1. Defining and validating subgroup quality (Q(s)), a metric that quantifies the viability of a symbol-to-symbol shortcut.\n\n2. Uncovering a distinct U-shaped accuracy pattern in multi-digit multiplication, which perfectly mirrors the quality of the simplest, lowest-token subgroups. This alignment serves as the \"smoking gun\" evidence that LMs utilize a \"fewest-tokens-first\" learning heuristic, demonstrating they learn a hierarchy of symbol-to-symbol mappings instead of the multiplication algorithm.\n\n3. Demonstrating that LMs achieve high accuracy on terminal output digits through pattern recognition, potentially by memorizing frequent digit combinations.\n\n4. Showing that improved ability to identify partial products (an inductive step) does not translate deductively to improved task learning, suggesting the gains stem from the symbol learning process rather than intrinsic computational methods.\n\nKey Strengths: The framework provides a computable lens for analyzing shortcut viability under token constraints. The strong empirical evidence confirms that LMs are driven by greedy symbolic learning rather than procedural computation.\n\nKey Weaknesses: Experiments suggest that while LMs improve in identifying inductive sub-steps, this ability does not deductively translate to improved overall task learning, implying the observed gains are from symbolic pattern recognition rather than genuine internal computation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The central claim that LMs are greedy symbolic learners exploiting shortcuts is supported by the convincing correlation between the U-shaped position-level accuracy observed in models (Gemma-2-2B and Llama-3.1-8B) and the calculated subgroup quality (Q(s)) of the simplest, lowest-token subgroups.\nThe high accuracy at the edges (first and last digits) is specifically explained by the existence of \"cheap but good\" rules (low-token shortcuts) for those positions, while middle digits require \"expensive\" rules combining more digits, which the model learns later.\nFurthermore, the observation that LMs struggle to translate improved recognition of partial products into better overall task performance confirms that their learning is focused on the symbol learning process rather than a true procedural algorithm.", "audience_yn": "Yes", "audience_explanation": "TMLR’s audience is highly interested in foundational machine learning mechanisms and the limitations of modern neural architectures, particularly in the context of trustworthy AI and reasoning.\nThe paper directly addresses a fundamental question concerning the nature of intelligence in large language models: do they compute or pattern match?\nThe introduction of subgroup induction provides a practical adaptation of the high-level theoretical framework of Solomonoff Induction (SI) to modern LLMs, which is relevant to researchers studying universal prediction and algorithmic probability.\nFurthermore, the empirical discovery of the U-shaped learning dynamic fundamentally challenges prior assumptions about how transformers acquire arithmetic skills.\nThe findings related to using subgroup entropy to quantify the symbolic complexity of Chain-of-Thought (CoT) paths and predict performance provide a significant diagnostic tool for advancing complex reasoning research, a critical area for the TMLR community.", "requested_changes": "1. While the paper notes that Q(s) is computationally expensive and H(s) is better for error estimation, a deeper analysis comparing when Q(s) is necessary (e.g., for mechanistic interpretability of token usage) versus when H(s) suffices (e.g., for error bounding) would strengthen the discussion.\n\n2. It would be nice to see the comparison in Reasoning models like deepseek-reasoner.", "broader_impact_concerns": "This work focused on understanding the internal learning mechanisms of language models in a controlled setting (arithmetic) to improve model reliability and reasoning capability. The findings contribute positively to the pursuit of trustworthy and interpretable AI. The sources provided do not indicate any immediate negative ethical or societal implications that would require a dedicated Broader Impact Statement.", "additional_comments": "The subgroup induction framework offers a highly innovative and quantifiable approach to interpreting the learning strategies of LLMs. The finding that the U-shaped accuracy curve in multiplication is a direct consequence of LMs greedily exploiting simple, low-token shortcuts is a compelling piece of evidence that contributes significantly to the ongoing debate about algorithmic versus symbolic learning in neural networks.", "extra_scores": {}, "extra_text": {}}, {"review_id": "RP8k91pUA8", "reviewer": "Reviewer_YHD5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a task called \"subgroup induction,\" which is then used to test the capability of language models to learn arithmetic. However, I do not quite understand this paper. Everything I say in the review is an educated guess", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experiments seem extensive. The main argument is that LLMs learn simple shortcuts first before learning complex ones, this is intuitive and, perhaps, not a too surprising discovery", "audience_yn": "Yes", "audience_explanation": "It studies large language models", "requested_changes": "I think the basic concepts should be explained much better. I do not know Solomonoff induction. I also have not heard of universal predictors.\n\nIn section 1.1, it is quite unclear which part comes from the author and which part comes from the original Solomonoff induction. The authors say that \"we formalize these components as follows\", but I thought this section is about introducing the prior literature, not about introducing their own formalization? What is prior knowledge and what is not? The complete lack of reference in this section is also a hint of this problem\n\nIn section 2, the problem of subgroup induction is introduced in an entirely random way. I do not see why this problem is interesting or important, why it is called \"subgroup\" and why it is called \"induction.\" To an ordinary reader, it feels like the authors are just making random things up. It feels that the authors are studying a random artificial problem of unclear relevance. Again, the lack of reference in this section is a hint of this problem", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)