Does Math Reasoning Improve General LLM Capabilities? Understanding
  Transferability of LLM Reasoning
Authors: name: Maggie Huan; name: Yuetai Li; name: Tuney Zheng; name: Xiaoyu Xu; name: Seungone Kim; name: Minxin Du; name: Radha Poovendran; name: Graham Neubig; name: Xiang Yue
Published: 2025-07-01 05:23:05

Abstract: Math reasoning has become the poster child of progress in large language
models (LLMs), with new models rapidly surpassing human-level performance on
benchmarks like MATH and AIME. But as math leaderboards improve week by week,
it is worth asking: do these gains reflect broader problem-solving ability or
just narrow overfitting? To answer this question, we evaluate over 20
open-weight reasoning-tuned models across a broad suite of tasks, including
math, scientific QA, agent planning, coding, and standard
instruction-following. We surprisingly find that most models that succeed in
math fail to transfer their gains to other domains. To rigorously study this
phenomenon, we conduct controlled experiments on Qwen3-14B models using
math-only data but different tuning methods. We find that reinforcement
learning (RL)-tuned models generalize well across domains, while supervised
fine-tuning (SFT)-tuned models often forget general capabilities. Latent-space
representation and token-space distribution shift analyses reveal that SFT
induces substantial representation and output drift, while RL preserves
general-domain structure. Our results suggest a need to rethink standard
post-training recipes, particularly the reliance on SFT-distilled data for
advancing reasoning models.