Language Models are Symbolic Learners in Arithmetic
Authors: name: Chunyuan Deng; name: Zhiqi Li; name: Roy Xie; name: Ruidi Chang; name: Hanjie Chen
Published: 2024-10-21 01:57:16

Abstract: Large Language Models (LLMs) are thought to struggle with arithmetic learning
due to the inherent differences between language modeling and numerical
computation, but concrete evidence has been lacking. This work responds to this
claim through a two-side experiment. We first investigate whether LLMs leverage
partial products during arithmetic learning. We find that although LLMs can
identify some partial products after learning, they fail to leverage them for
arithmetic tasks, conversely. We then explore how LLMs approach arithmetic
symbolically by breaking tasks into subgroups, hypothesizing that difficulties
arise from subgroup complexity and selection. Our results show that when
subgroup complexity is fixed, LLMs treat a collection of different arithmetic
operations similarly. By analyzing position-level accuracy across different
training sizes, we further observe that it follows a U-shaped pattern: LLMs
quickly learn the easiest patterns at the first and last positions, while
progressively learning the more difficult patterns in the middle positions.
This suggests that LLMs select subgroup following an easy-to-hard paradigm
during learning. Our work confirms that LLMs are pure symbolic learners in
arithmetic tasks and underscores the importance of understanding them deeply
through subgroup-level quantification.