Two Stories in Understanding Arithmetic Learning at Scale

Authors: Chunyuan Deng, Zhiqi Li, Roy Xie, Ruidi Chang, Hanjie Chen
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Math Arithmetic, Mechanistic Understanding, Large Language Models

Abstract:
Large Language Models (LLMs) have long been suspected of struggling with arithmetic learning due to the inherent differences between language modeling and numerical calculation, but concrete evidence for this claim has been lacking. In this work, we respond to this claim through a two-side experiment. First, we investigate whether LLMs leverage partial products during arithmetic learning. We find that although LLMs can identify some partial products after learning, they fail to leverage them during the learning process. Instead, these partial products appear to be by-products from the pattern fitting. Second, we examine whether LLMs treat arithmetic in a purely symbolic manner. We decompose the task into subgroup-level (pairs of subportion tokens), hypothesizing that arithmetic learning difficulty arises from subgroup complexity, subgroup selection, and sequential prediction. We find that, given controlled subgroup complexity, LLMs treat a collection of different arithmetic operations similarly. Furthermore, arithmetic with low entropy in the subgroup label space lends to be more learnable. By analyzing position-level accuracy across different training sizes, we further explore subgroup selection. We observe that position-level accuracy follows a U-shaped pattern: LLMs quickly learn the easiest patterns at the first and last positions, while progressively learning the more difficult patterns in the middle positions. This phenomenon suggests that LLMs follow an easy-to-hard learning mechanism based on subgroup complexity. Our work confirms that LLMs are pure symbolic learners in arithmetic tasks and underscores the importance of understanding them deeply through subgroup-level quantification.

Decision: reject

Reviews and discussion:

Metareview: